Added ObjectNav task definition, dataset, metrics, goal sensor #284

mathfac · 2020-01-27T02:58:28Z

Motivation and Context

Added ObjectNav task definition, dataset, metrics, goal sensor.

ObjectGoal Nav Task definition: metrics, goals, sensors with documentations.
ObjectGoal Nav Dataset definition and loading and link for downloading.
Topdown target objects, view points, bounding boxes visualization
Compatible with Benchmark API
Added ObjectNav DDPPO config for a baseline
Made PointNavDataset general enough to be inherited by other Dataset types
Refactored RLEnv to use measures for reward and success to speed up distance calculations
Added mechanism for Measures (Metrics) classes to be depend on other measure. That increased speed and calculation and decrease copy pasta. Speed improvements 2x: from 8 FPS to 15.5 FPS, compare to 40 FPS for PointGoal with same setup.
made SPL dependent on DistanceToGoal
added metrics calc to reset method for SPL and DistanceToGoal

with @Skylion007, inspired by @erikwijmans's an initial implementation.

How Has This Been Tested

Added the unit test.
Enable the task config for other general tests we have

Examples of the Object Goal Navigation Episodes:
Goal semantic category: chair

Goal semantic category: picture

Goal semantic category: sink

The target object is highlighted by red overlay on RGB image.

Map colors:
Red is a center of the target objet.
Green is an axes aligned bounding box of the target objet.
Red overlay zone near the target objects is view points/success zone.

Types of changes

Docs change / refactoring / dependency upgrade
New feature (non-breaking change which adds functionality)

configs/tasks/obj_nav_mp3d.yaml

erikwijmans · 2020-01-27T17:20:17Z

configs/tasks/obj_nav_mp3d.yaml

+  MAX_EPISODE_STEPS: 750
+SIMULATOR:
+  TURN_ANGLE: 30
+  TILT_ANGLE: 30


We are planning to add look_up and look_down actions to the action space for the challenge? If so, we should add a max tilt angle parameter.

We are planning to add look_up and look_down actions to the action space for the challenge?

Yes, so the agent can see objects hanging on the wall or placed on the floor.

Ah, any suggestion for the max title angle then?

-90..+90 degrees?

configs/tasks/obj_nav_mp3d.yaml

habitat/tasks/nav/nav.py

habitat/tasks/nav/object_nav_task.py

habitat/tasks/nav/nav.py

habitat/datasets/object_nav/object_nav_dataset.py

smorad · 2020-01-30T10:07:06Z

This is relevant to my interests, is there an associated dataset for this?

dhruvbatra · 2020-01-30T16:52:57Z

Dataset and challenge details for CVPR20 challenge/workshop are coming soon.

mathfac · 2020-01-31T02:14:16Z

Address the comments, refactored drawing functions, update the config. PR depends on #290.

codecov-io · 2020-01-31T22:42:59Z

Codecov Report

Merging #284 into master will increase coverage by 0.28%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master     #284      +/-   ##
==========================================
+ Coverage   77.05%   77.33%   +0.28%     
==========================================
  Files         108      108              
  Lines        7077     7086       +9     
==========================================
+ Hits         5453     5480      +27     
+ Misses       1624     1606      -18

Impacted Files	Coverage Δ
...abitat/sims/habitat_simulator/habitat_simulator.py	`91.85% <ø> (-0.04%)`	⬇️
examples/shortest_path_follower_example.py	`97.05% <ø> (ø)`	⬆️
habitat/core/utils.py	`51.31% <ø> (-0.64%)`	⬇️
habitat/tasks/nav/object_nav_task.py	`59.25% <0%> (ø)`	⬆️
test/test_sensors.py	`96.29% <100%> (ø)`	⬆️
habitat_baselines/common/base_trainer.py	`73.91% <100%> (+0.28%)`	⬆️
habitat_baselines/common/environments.py	`100% <100%> (+4%)`	⬆️
habitat/datasets/pointnav/pointnav_dataset.py	`81.94% <100%> (ø)`	⬆️
habitat/version.py	`100% <100%> (ø)`	⬆️
habitat/utils/visualizations/maps.py	`100% <100%> (ø)`	⬆️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2041054...812df92. Read the comment docs.

smorad · 2020-02-03T10:48:09Z

Does "compatible with benchmark api" refer to habitat_baselines? If so, could you include the baseline .yaml you created for testing purposes?

abhiskk · 2020-02-04T00:47:49Z

configs/tasks/objectnav_mp3d.yaml

+    WIDTH: 640
+    HEIGHT: 480
+    HFOV: 79
+    MIN_DEPTH: 0.5


Is there a reason for using a MIN_DEPTH of 0.5, the default config uses a value of 0?

Yes, that is motivated by real depth camera.

abhiskk · 2020-02-04T00:52:34Z

habitat/tasks/nav/nav.py

@@ -559,51 +565,95 @@ def get_original_map(self):

        return top_down_map

-    def draw_source_and_target(self, episode):


Just to double check, what exactly is the reason for getting rid of this method. From what I can tell by default the target point is drawn always and there is a config parameter to control drawing of the source point.

It was broke into _draw_point and _draw_goals_positions as we start drawing more sophisticated goals.

habitat/core/utils.py

abhiskk · 2020-02-04T00:58:33Z

habitat/core/utils.py

+    """
+
+    # Version of JSON library that encoder is compatible with.
+    __version__ = "2.0.9"


How is this __version__ being used, maybe rename it to version_compatible, __version__ is mostly used to indicate the version of package (which in this case is habitat).

abhiskk · 2020-02-04T01:01:10Z

habitat/datasets/object_nav/object_nav_dataset.py

+        )
+
+        assert set(self.category_to_task_category_id.keys()) == set(
+            self.category_to_mp3d_category_id.keys()


Should we have Matterport3d (mp3d) specific variable names here, is this implementation dataset (Matterport3D, Gibson) agnostic?

That's good question. We can change category_to_mp3d_category_id to category_to_scene_annotation_category_id and then use scene dataset type for further category conversions. cc @Skylion007

abhiskk · 2020-02-04T01:02:59Z

habitat/tasks/nav/nav.py

@@ -614,7 +664,7 @@ def reset_metric(self, *args: Any, episode, **kwargs: Any):
                    self._coordinate_min,
                    self._coordinate_max,
                    self._map_resolution,
-                )[::-1]


What is the reason for getting rid of the list reversal here?

Because, maps.draw_path method was changed that switching x and y isn't needed anymore. That makes maps.draw_path easier to use.

abhiskk · 2020-02-04T01:04:47Z

habitat/utils/visualizations/maps.py

@@ -47,6 +49,8 @@
 TOP_DOWN_MAP_COLORS[MAP_SOURCE_POINT_INDICATOR] = [0, 0, 200]
 TOP_DOWN_MAP_COLORS[MAP_TARGET_POINT_INDICATOR] = [200, 0, 0]
 TOP_DOWN_MAP_COLORS[MAP_SHORTEST_PATH_COLOR] = [0, 200, 0]
+TOP_DOWN_MAP_COLORS[MAP_VIEW_POINT_INDICATOR] = [245, 150, 150]


nit: if possible add color names here, it would be useful for someone reading the codebase.

mathfac · 2020-02-06T00:26:24Z

Added ObjectNav DDPPO config for a baseline cc @smorad
Updated the config to converged version for ObjectNav
Made PointNavDataset general enough to be inherited
Refactored RLEnv to use measures for reward and success to speed up distance calculations
Added mechanisms for measures to depend on other measure. That increased speed and calculation and decrease copy pasta. Speed improvements 2x: from 8 FPS to 15.5 FPS, compare to 40 FPS for PointGoal with same setup.
made SPL dependent on DistanceToGoal
added metrics calc to reset method for SPL and DistanceToGoal

…endent on DistToGoal, added metrics calc to reset method

…oning from ObjectNavDataset

mathfac · 2020-02-06T03:26:25Z

Added doc to FloatJSON encoder,
Added color names
Removed mp3d mentioning from ObjectNavDataset class

erikwijmans · 2020-02-06T18:30:38Z

habitat_baselines/config/objectnav/ddppo_objectnav.yaml

+VIDEO_OPTION: ["disk", "tensorboard"]
+TENSORBOARD_DIR: "/checkpoint/maksymets/logs/habitat_baselines/ddppo/obj_nav_mp3d"
+VIDEO_DIR: "data/video_dir"
+TEST_EPISODE_COUNT: 2


Let's put the actual number of val episodes here. It is hard to figure out what number should go here when you want to evaluate on everything otherwise.

abhiskk

LGTM 👍

…ookresearch#284) Added ObjectNav task definition, dataset, metrics, goal sensor. ObjectGoal Nav Task definition: metrics, goals, sensors with documentations. ObjectGoal Nav Dataset definition and loading and link for downloading. Topdown target objects, view points, bounding boxes visualization Compatible with Benchmark API Added ObjectNav DDPPO config for a baseline Made PointNavDataset general enough to be inherited by other Dataset types Refactored RLEnv to use measures for reward and success to speed up distance calculations Added mechanism for Measures (Metrics) classes to be depend on other measure. That increased speed and calculation and decrease copy pasta. Speed improvements 2x: from 8 FPS to 15.5 FPS, compare to 40 FPS for PointGoal with same setup. made SPL dependent on DistanceToGoal added metrics calc to reset method for SPL and DistanceToGoal

mathfac requested a review from erikwijmans January 27, 2020 02:58

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Jan 27, 2020

mathfac requested review from abhiskk and msbaines January 27, 2020 02:58

mathfac force-pushed the object_nav_task branch from 7a81e9f to 0b8ecc6 Compare January 27, 2020 03:03

erikwijmans reviewed Jan 27, 2020

View reviewed changes

mathfac added 5 commits January 30, 2020 13:58

Added geodesic distance for multiple goals

ec6c1df

Added geodesic distance for multiple goals

7a2a484

Added ObjectNav task definition, dataset, metrics, goal sensor

1727467

Added geodesic distance for multiple goals

7a0a90e

Refactored drawing functions, updated ObjectGoal Nav config

df8ba3d

mathfac force-pushed the object_nav_task branch from 0b8ecc6 to df8ba3d Compare January 31, 2020 02:11

mathfac force-pushed the object_nav_task branch from 27d8546 to 20a21bc Compare January 31, 2020 18:55

Updated the configs

8dd1945

mathfac force-pushed the object_nav_task branch from 20a21bc to 8dd1945 Compare January 31, 2020 22:29

mathfac added 2 commits February 2, 2020 19:56

Added dataset link, unified version of the task.

2041054

Merge branch 'master' into object_nav_task

efa2cda

abhiskk reviewed Feb 4, 2020

View reviewed changes

Made PointNavDataset general enough to be inherited

2a7bb2a

Refactored RLEnv to use measures for reward and success, made SPL dep…

bcfb573

…endent on DistToGoal, added metrics calc to reset method

mathfac force-pushed the object_nav_task branch from 07c5aaf to bcfb573 Compare February 6, 2020 00:33

Added doc to FloatJSON encoder, added color names, removed mp3d menti…

4d7d8b3

…oning from ObjectNavDataset

Fixed for sensors test

c928734

mathfac force-pushed the object_nav_task branch from 1708855 to c928734 Compare February 6, 2020 04:34

erikwijmans reviewed Feb 6, 2020

View reviewed changes

mathfac force-pushed the object_nav_task branch from 5e6c130 to c105390 Compare February 6, 2020 20:54

Switched code to heading and GPS sensors using single array wrapping.

1e4563d

mathfac force-pushed the object_nav_task branch from c105390 to 1e4563d Compare February 6, 2020 22:08

abhiskk approved these changes Feb 6, 2020

View reviewed changes

Added number of episodes in val split

812df92

mathfac merged commit f5e29c6 into master Feb 7, 2020

mathfac deleted the object_nav_task branch February 7, 2020 19:05

erikwijmans mentioned this pull request Mar 12, 2020

Add success measure #332

Merged

dhruvbatra mentioned this pull request Apr 13, 2020

Image nav task #333

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added ObjectNav task definition, dataset, metrics, goal sensor #284

Added ObjectNav task definition, dataset, metrics, goal sensor #284

mathfac commented Jan 27, 2020 •

edited

Loading

erikwijmans Jan 27, 2020

dhruvbatra Jan 27, 2020

erikwijmans Jan 27, 2020

mathfac Jan 28, 2020

smorad commented Jan 30, 2020

dhruvbatra commented Jan 30, 2020

mathfac commented Jan 31, 2020

codecov-io commented Jan 31, 2020 •

edited by codecov bot

Loading

smorad commented Feb 3, 2020

abhiskk Feb 4, 2020

mathfac Feb 5, 2020

abhiskk Feb 4, 2020

mathfac Feb 5, 2020

abhiskk Feb 4, 2020

abhiskk Feb 4, 2020

mathfac Feb 5, 2020

abhiskk Feb 4, 2020

mathfac Feb 5, 2020

abhiskk Feb 4, 2020

mathfac commented Feb 6, 2020 •

edited

Loading

mathfac commented Feb 6, 2020

erikwijmans Feb 6, 2020

abhiskk left a comment

		@@ -559,51 +565,95 @@ def get_original_map(self):

		return top_down_map

		def draw_source_and_target(self, episode):

Added ObjectNav task definition, dataset, metrics, goal sensor #284

Added ObjectNav task definition, dataset, metrics, goal sensor #284

Conversation

mathfac commented Jan 27, 2020 • edited Loading

Motivation and Context

How Has This Been Tested

Types of changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smorad commented Jan 30, 2020

dhruvbatra commented Jan 30, 2020

mathfac commented Jan 31, 2020

codecov-io commented Jan 31, 2020 • edited by codecov bot Loading

Codecov Report

smorad commented Feb 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mathfac commented Feb 6, 2020 • edited Loading

mathfac commented Feb 6, 2020

Choose a reason for hiding this comment

abhiskk left a comment

Choose a reason for hiding this comment

mathfac commented Jan 27, 2020 •

edited

Loading

codecov-io commented Jan 31, 2020 •

edited by codecov bot

Loading

mathfac commented Feb 6, 2020 •

edited

Loading