Reproducing Results on the https://www.visuallocalization.net/details/17831/ Benchmark #20

fulkast · 2021-11-25T06:09:27Z

Thank you for sharing your implementation and the tools surrounding this localization framework!

I have been trying to reproduce the results of hloc + pixloc on the visual localization benchmark for the Extended CMU dataset. However, I haven't been able to get results close to the values seen on the linked benchmark. The values I'm currently getting are:

Locally, I've downloaded the pixloc_cmu pre-trained weights hosted here, and I'm running the following command:
python -m pixloc.run_CMU --from_poses
Which after hours of running terminates with the message (truncated):
[11/25/2021 00:32:21 pixloc INFO] Finished evaluating all slices, you can now submit the file /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice2-3-4-5-6-13-14-15-16-17-18-19-20-21.txt to https://www.visuallocalization.net/submission/
I'm assuming that --from_poses runs the evaluation using hloc poses as a start, is this correct? Also, do you have any pointers on what I must be doing wrong?

The text was updated successfully, but these errors were encountered:

patelajaychh · 2021-11-26T06:20:34Z

$  python -m pixloc.run_Cambridge

Std Output - ( Only for KingsCollege scene )

[11/26/2021 18:46:06 pixloc INFO] Parsed configuration:
experiment: pixloc_megadepth
features: {}
optimizer:
  num_iters: 100
  pad: 2
refinement:
  num_dbs: 5
  multiscale:
  - 4
  - 1
  point_selection: all
  normalize_descriptors: true
  average_observations: true
  filter_covisibility: false
  do_pose_approximation: false

[11/26/2021 18:46:06 pixloc INFO] Working on scene KingsCollege.
[11/26/2021 18:46:06 pixloc.localization.model3d INFO] Reading COLMAP model /home/ajay/pixloc/outputs/hloc/Cambridge/KingsCollege/sfm_superpoint+superglue.
[11/26/2021 18:46:08 pixloc.utils.io INFO] Imported 343 images from query_list_with_intrinsics.txt
[11/26/2021 18:46:08 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/26/2021 18:46:11 pixloc.localization.localizer INFO] Starting the localization process...
 47%|██████████████████████████████████████████▉                                                | 162/343 [12:05<11:33,  3.83s/it][11/26/2021 18:58:18 pixloc.localization.base_refiner INFO] Optimization failed for query seq2/frame00033.png
100%|███████████████████████████████████████████████████████████████████████████████████████████| 343/343 [23:46<00:00,  4.16s/it]
[11/26/2021 19:09:57 pixloc.utils.io INFO] Writing the localization results to /home/ajay/pixloc/outputs/results/pixloc_Cambridge_KingsCollege.txt.
[11/26/2021 19:09:57 pixloc INFO] Evaluate scene KingsCollege: /home/ajay/pixloc/outputs/results/pixloc_Cambridge_KingsCollege.txt
[11/26/2021 19:09:57 pixloc.utils.eval INFO]
Median errors: 0.128m, 0.228deg
Percentage of test images localized within:
        1cm, 1deg : 0.29%
        2cm, 2deg : 1.46%
        3cm, 3deg : 4.37%
        5cm, 5deg : 14.29%
        25cm, 2deg : 73.47%
        50cm, 5deg : 87.76%
        500cm, 10deg : 96.21%

Here the med errors are 12.8cm/0.228deg which are different from what is claimed in paper ( 14cm/0.24deg, 13cm/0.24deg with oracle prior ). Please review this and help us to understand reason for this deviation.

Similarly, 7Scenes results are also different than claimed.

sarlinpe · 2021-11-27T03:43:14Z

@fulkast: This is indeed abnormal, let me run this again and dig into it. Did you also run the evaluation without --from_poses? if yes, what results do you obtain? With the recent code cleanup the recall should be around:

88.9 / 91.0 / 94.6	81.1 / 82.4 / 85.6	62.4 / 64.2 / 69.5

Which is slightly higher than what is reported in the original paper.

@patelajaychh Please don't hijack this issue, instead open a new one to discuss the results of Cambridge/7Scenes. The README explicitly mentions that the results might differ slightly, and generally improve rather than get worse. Given the rounding, the results that you obtain are rather consistent. Does it consistently get worse on other scenes? What is the difference observed with 7Scenes?

fulkast · 2021-11-27T12:08:56Z

Hi, @skydes. Yes, I've also run the evaluation without the flag --from_poses and the results are on the third last row of the image from my previous post:

31.4/60.5/94.0	0.6/3.3/ 69.2	0.4/2.2/55.3

Does this slice list look correct for the evaluation on the Extended CMU dataset: pixloc_CMU_slice2-3-4-5-6-13-14-15-16-17-18-19-20-21.txt?

sarlinpe · 2021-11-28T09:09:59Z

On my end running python -m pixloc.run_CMU --slices 2-6 gives 89.0 / 91.1 / 94.6 for the urban category, consistent with my previous message. The slice list that you report is fine.

Can you dump the versions of all the main libraries? torch, numpy, opencv
Can you share the full console output, including the progress bars?

fulkast · 2021-11-28T14:19:02Z

Thanks for checking this! I will re-run the experiment and report back.

fulkast · 2021-11-29T09:35:01Z

Hi @skydes,

Following your suggestion, I've re-run the evaluation on the urban category and got the results: 31.5 / 60.7 / 94.1 (pretty close to the results I previously reported). Below is the information that you asked for, please let me know if anything looks unexpected.

Pip freeze on the main libraries shows the following:
numpy==1.17.4
opencv-python==4.5.4.58
torch==1.10.0+cu113
I will also attach the full pip freeze output here.
Here's the full output of my run on the urban category:

python3 -m pixloc.run_CMU --slices 2-6  experiment=pixloc_author_reference > evaluation_run.txt
[11/28/2021 23:02:54 pixloc INFO] Parsed configuration:
experiment: pixloc_author_reference
features: {}
optimizer:
  num_iters: 100
  pad: 2
refinement:
  num_dbs: 2
  point_selection: all
  normalize_descriptors: true
  average_observations: false
  filter_covisibility: false
  do_pose_approximation: false

[11/28/2021 23:02:54 pixloc INFO] Working on slice 2.
[11/28/2021 23:02:55 pixloc.localization.model3d INFO] Reading COLMAP model /home/frank/github/pixloc/outputs/hloc/CMU/slice2/sfm_superpoint+superglue/model.
[11/28/2021 23:02:56 pixloc.utils.io INFO] Imported 3655 images from queries_with_intrinsics.txt
[11/28/2021 23:02:56 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/28/2021 23:03:01 pixloc.localization.localizer INFO] Starting the localization process...
100%|█████████████████████████████████████████████████████████████████| 3655/3655 [1:11:41<00:00,  1.18s/it]
[11/29/2021 00:14:42 pixloc.utils.io INFO] Writing the localization results to /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice2.txt.
[11/29/2021 00:14:43 pixloc INFO] Working on slice 3.
[11/29/2021 00:14:43 pixloc.localization.model3d INFO] Reading COLMAP model /home/frank/github/pixloc/outputs/hloc/CMU/slice3/sfm_superpoint+superglue/model.
[11/29/2021 00:14:44 pixloc.utils.io INFO] Imported 5074 images from queries_with_intrinsics.txt
[11/29/2021 00:14:44 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/29/2021 00:14:49 pixloc.localization.localizer INFO] Starting the localization process...
100%|█████████████████████████████████████████████████████████████████| 5074/5074 [1:42:21<00:00,  1.21s/it]
[11/29/2021 01:57:11 pixloc.utils.io INFO] Writing the localization results to /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice3.txt.
[11/29/2021 01:57:12 pixloc INFO] Working on slice 4.
[11/29/2021 01:57:12 pixloc.localization.model3d INFO] Reading COLMAP model /home/frank/github/pixloc/outputs/hloc/CMU/slice4/sfm_superpoint+superglue/model.
[11/29/2021 01:57:13 pixloc.utils.io INFO] Imported 4728 images from queries_with_intrinsics.txt
[11/29/2021 01:57:13 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/29/2021 01:57:18 pixloc.localization.localizer INFO] Starting the localization process...
100%|█████████████████████████████████████████████████████████████████| 4728/4728 [1:30:27<00:00,  1.15s/it]
[11/29/2021 03:27:45 pixloc.utils.io INFO] Writing the localization results to /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice4.txt.
[11/29/2021 03:27:45 pixloc INFO] Working on slice 5.
[11/29/2021 03:27:45 pixloc.localization.model3d INFO] Reading COLMAP model /home/frank/github/pixloc/outputs/hloc/CMU/slice5/sfm_superpoint+superglue/model.
[11/29/2021 03:27:46 pixloc.utils.io INFO] Imported 2369 images from queries_with_intrinsics.txt
[11/29/2021 03:27:46 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/29/2021 03:27:49 pixloc.localization.localizer INFO] Starting the localization process...
100%|███████████████████████████████████████████████████████████████████| 2369/2369 [44:19<00:00,  1.12s/it]
[11/29/2021 04:12:08 pixloc.utils.io INFO] Writing the localization results to /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice5.txt.
[11/29/2021 04:12:08 pixloc INFO] Working on slice 6.
[11/29/2021 04:12:08 pixloc.localization.model3d INFO] Reading COLMAP model /home/frank/github/pixloc/outputs/hloc/CMU/slice6/sfm_superpoint+superglue/model.
[11/29/2021 04:12:10 pixloc.utils.io INFO] Imported 2547 images from queries_with_intrinsics.txt
[11/29/2021 04:12:10 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[11/29/2021 04:12:15 pixloc.localization.localizer INFO] Starting the localization process...
100%|███████████████████████████████████████████████████████████████████| 2547/2547 [51:30<00:00,  1.21s/it]
[11/29/2021 05:03:45 pixloc.utils.io INFO] Writing the localization results to /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice6.txt.
[11/29/2021 05:03:45 pixloc INFO] Finished evaluating all slices, you can now submit the file /home/frank/github/pixloc/outputs/results/pixloc_CMU_slice2-3-4-5-6.txt to https://www.visuallocalization.net/submission/

For more context, I'm running on commit d5f7f7d02910b9ac4c0289360c1, and here's a snapshot of my git log:

sarlinpe · 2021-11-29T18:16:54Z

What is the pixloc_author_reference experiment? Does it mean that you are evaluating weights that you retrained rather than the pretrained model? This was not clear from your description. If yes, can you share the training curves? do they look consistent with those of our reference experiment? Are you able to visualize the predictions on CMU in the demo notebook?

fulkast · 2021-11-30T00:38:55Z

Hi! Sorry, I should have elaborated on the last post. pixloc_author_reference is a folder I created to store the downloaded pre-trained weights, retrieved from here. So here's my outputs directory structure, where pixloc_cmu is my training experiment and pixloc_author_reference only has the one downloaded checkpoint_best.tar:

To double-check, here's the sha1sum of the checkpoint_best.tar file that I downloaded:

sha1sum code: 0f33613a21bcb898dfdf469273d6a20ed7dc15f9

That said, I will share the loss curves from my training experiment (the one called pixloc_cmu), in case there is something anomalous there.

And, for completeness, here's the tf-events file:
events.out.tfevents.1637452030.the-machine.5052.zip

Please let me know in case anything looks suspicious.

[Edit]:

If possible, may I ask what metrics you get when you run python -m pixloc.run_CMU --slices 2. By isolating it to slice 2, I will be able to iterate faster in trying to find the error on my end.

Best,
Frank

ccy-ustb · 2021-12-02T06:33:55Z

hello，@fulkast
when i run “python3 -m pixloc.run_Aachen --from_poses” meet an error：

caochengyang@caochengyang-Lenovo-Legion-R9000P2021H:~/pixloc$ python3 -m pixloc.run_Aachen --from_poses
True
[12/02/2021 10:54:15 pixloc INFO] Parsed configuration:
experiment: pixloc_megadepth
features:
preprocessing:
resize: 1600
optimizer:
num_iters: 50
pad: 1
refinement:
num_dbs: 5
min_points_opt: 100
point_selection: inliers
normalize_descriptors: true
average_observations: true
layer_indices:

0
1
[12/02/2021 10:54:15 pixloc.localization.model3d INFO] Reading COLMAP model /home/caochengyang/pixloc/outputs/hloc/Aachen/sfm_superpoint+superglue.
[12/02/2021 10:54:27 pixloc.utils.io INFO] Imported 824 images from day_time_queries_with_intrinsics.txt
[12/02/2021 10:54:27 pixloc.utils.io INFO] Imported 98 images from night_time_queries_with_intrinsics.txt
[12/02/2021 10:54:27 pixloc.pixlib.utils.experiments INFO] Loading checkpoint checkpoint_best.tar
[12/02/2021 10:54:32 pixloc.localization.localizer INFO] Reading hloc logs...
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/caochengyang/pixloc/pixloc/run_Aachen.py", line 77, in
main()
File "/home/caochengyang/pixloc/pixloc/run_Aachen.py", line 66, in main
localizer = PoseLocalizer(paths, conf)
File "/home/caochengyang/pixloc/pixloc/localization/localizer.py", line 145, in init
with open(paths.log_path, 'rb') as f:
AttributeError: 'Paths' object has no attribute 'log_path'

Have you encountered it? Can you help me solve it? grateful！！！

fulkast · 2021-12-02T15:38:06Z

@caochengyang0828 I will respond to the new issue you started.

sarlinpe · 2021-12-03T15:29:50Z

@fulkast This all looks good, my setup seems to match on every point, so I really have no idea of what's happening here. Here is what I get for slice 2 only: 17.6 / 17.9 / 19.2

fulkast · 2021-12-06T10:36:48Z

Thanks, @skydes. Yeah, I get 7.3 / 13.1 / 19.2 for using only slice 2. Would it be okay to share the pixloc_CMU_slice2.txt file you used for the slice 2 submission? I want to take a closer look at my culprit cases. If you'd prefer to send them privately, this is my email: [email protected]

VictorZoo · 2021-12-07T14:23:31Z

Got the same situation with you. And got the simiar results to yours. python -m pixloc.run_CMU

sarlinpe · 2021-12-08T08:15:15Z

Results for Slice 2: pixloc_CMU_slice2.txt
Output of pip freeze: pip_freeze.txt

Again, this is obtained by running the following on a fresh clone of the repo:

pip3 install -r requirements.txt
python -m pixloc.download --select CMU checkpoints --CMU_slices 2
python -m pixloc.run_CMU --slices 2 --results outputs/results/pixloc_CMU_slice2.txt

fulkast · 2021-12-08T14:14:28Z

Hi @skydes,
Thanks again for looking into this. While I don't yet have a sense of the cause of the problem, I can now at least reproduce your results on one of my machines.

Both authors_pixloc_slice_2_6_home_device and authors_pixloc_slice_2_6_office_device were generated using the same command python3 -m pixloc.run_CMU --slices 2-6 experiment=pixloc_author_reference with the same setup (at least what I believe to be the same. Clearly they aren't), on my home and work computers respectively. Next, I will take a deeper look at the different between the two setups and report my findings.

VictorZoo · 2021-12-09T02:54:21Z

Looking forward to the final solution to this problem that has been bothering me for a long time. T.T Thank you very much @fulkast

ccy-ustb · 2021-12-13T11:10:39Z

hello,@fulkast,@skydes.
After i update the repository,on 7Scenes datasets,i got the similar results with paper.
but on CMU datasets,i run :

pip3 install -r requirements.txt
python3 -m pixloc.download --select CMU checkpoints --CMU_slices 2
python3 -m pixloc.run_CMU --slices 2 --results outputs/results/pixloc_CMU_slice2.txt

I still get this result：

Is this the reason for relying on the library version?
This is my output of pip freeze:
pip_freeze (1).txt

Thank you so much for answering this question, it has bothered me for a long time！

fulkast · 2021-12-26T21:52:44Z

Happy holidays all! I am away from work now and won't be making physical progress on this issue. In the meantime, however, I did come across this pytorch issue that looks like it might be related pytorch/pytorch#70162. I will report more on this in the new year.

fulkast · 2022-02-17T13:12:55Z

Hi all! A happy new year to you all and I've got potentially some good news :)

In implementing some local tests I noticed that the results of my tensor matmul operations can be significantly different, depending on whether they were run on the GPU or the CPU. Long story short, this led me to this note here:
https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices which was also relevant to this PyTorch issue: pytorch/pytorch#67185

After setting

torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.allow_tf32 = False

at the top of run_CMU.py my tests on a small subset of the data is looking more consistent with what @skydes gets on slice 2 of the Extended CMU Seasons dataset. I'm now running on the full dataset and will report back soon.

fulkast · 2022-02-18T01:32:27Z

Hi @skydes

I'm happy to share that I've been able to reproduce your results on slice 2 and I'm hence confident that I should be able to reproduce the results for the rest of the dataset as well.

It came down to the pytorch issue mentioned in this link. Essentially, I was already getting positional Euclidean errors in transforming 3-D points to the camera frame on the order of 10cm because I by default torch.backends.cuda.matmul.allow_tf32 = True on my RTX GPU.

Thank you for your feedback on my questions; your data was helpful in letting me hone in on the source of the problem!

Best,
Frank

VictorZoo mentioned this issue Dec 7, 2021

Different results on Evaluation with the same pretrained model #25

Open

fulkast closed this as completed Feb 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reproducing Results on the https://www.visuallocalization.net/details/17831/ Benchmark #20

Reproducing Results on the https://www.visuallocalization.net/details/17831/ Benchmark #20

fulkast commented Nov 25, 2021 •

edited

Loading

patelajaychh commented Nov 26, 2021

sarlinpe commented Nov 27, 2021 •

edited

Loading

fulkast commented Nov 27, 2021 •

edited

Loading

sarlinpe commented Nov 28, 2021 •

edited

Loading

fulkast commented Nov 28, 2021

fulkast commented Nov 29, 2021 •

edited

Loading

sarlinpe commented Nov 29, 2021

fulkast commented Nov 30, 2021 •

edited

Loading

ccy-ustb commented Dec 2, 2021

fulkast commented Dec 2, 2021

sarlinpe commented Dec 3, 2021

fulkast commented Dec 6, 2021

VictorZoo commented Dec 7, 2021

sarlinpe commented Dec 8, 2021

fulkast commented Dec 8, 2021

VictorZoo commented Dec 9, 2021 •

edited

Loading

ccy-ustb commented Dec 13, 2021

fulkast commented Dec 26, 2021 •

edited

Loading

fulkast commented Feb 17, 2022

fulkast commented Feb 18, 2022

Reproducing Results on the https://www.visuallocalization.net/details/17831/ Benchmark #20

Reproducing Results on the https://www.visuallocalization.net/details/17831/ Benchmark #20

Comments

fulkast commented Nov 25, 2021 • edited Loading

patelajaychh commented Nov 26, 2021

sarlinpe commented Nov 27, 2021 • edited Loading

fulkast commented Nov 27, 2021 • edited Loading

sarlinpe commented Nov 28, 2021 • edited Loading

fulkast commented Nov 28, 2021

fulkast commented Nov 29, 2021 • edited Loading

sarlinpe commented Nov 29, 2021

fulkast commented Nov 30, 2021 • edited Loading

ccy-ustb commented Dec 2, 2021

fulkast commented Dec 2, 2021

sarlinpe commented Dec 3, 2021

fulkast commented Dec 6, 2021

VictorZoo commented Dec 7, 2021

sarlinpe commented Dec 8, 2021

fulkast commented Dec 8, 2021

VictorZoo commented Dec 9, 2021 • edited Loading

ccy-ustb commented Dec 13, 2021

fulkast commented Dec 26, 2021 • edited Loading

fulkast commented Feb 17, 2022

fulkast commented Feb 18, 2022

fulkast commented Nov 25, 2021 •

edited

Loading

sarlinpe commented Nov 27, 2021 •

edited

Loading

fulkast commented Nov 27, 2021 •

edited

Loading

sarlinpe commented Nov 28, 2021 •

edited

Loading

fulkast commented Nov 29, 2021 •

edited

Loading

fulkast commented Nov 30, 2021 •

edited

Loading

VictorZoo commented Dec 9, 2021 •

edited

Loading

fulkast commented Dec 26, 2021 •

edited

Loading