Object detection | Keypoints to object detection model

[ALEXKIRNAS]

Hi,
@jch1 , @tombstone , @derekjchow , I would appreciate your help.

ADDING FUNCTIONALITY
I recently start to use SSD with MobileNet feature extractor for solving face detection problem . In term of this task I also need to to detect landmarks (keypoints). I saw that repo contain some files that connected with keypoints ( link ), but I don`t see how to add them to model. So can you describe how to do that?

With the best regards,
Alexander

[Sheepybloke2-0]

Any update on this? It seems that the visualizations there and the standard field is there, but I can't seem to get the keypoints from the graph (I'm using ssd_mobilenet_v1_coco). Currently failing when I'm calling keypoints = graph.get_tensor_by_name('detection_keypoints:0')

[wcastil]

@pkulzc I saw you had done most of the work on adding keypoints to the detection framework. Any chance you can give an update on if/when that functionality will be available?

[dougsouza]

As far as I've got in the code I saw that only MaskRCNNBoxPredictor supports keypoint prediction. How hard it would be to add keypoint prediction to a ConvolutionalBoxPredictor as well? Any thoughts?

[dustindorroh]

I had the same issue, but got reading and writing tfrecords to work. I have links to the source that helped me understand but, here is the tl,dr

Short Answer

1. When creating tfrecords write the keypoints to 'image/object/keypoint/x' and 'image/object/keypoint/y'
2. When reading tfrecords use 'groundtruth_keypoints' key
3. Trying to use higher level inputs.create_train_input_fn will not load keypoints because a few levels down when tf_example_decoder.TfExampleDecoder is called it isn't called with num_keypoints which defaults to 0 and thus never runs decoding.
4. Example of writing keypoints tfrecords . A little messy, but eh not sure if this topic is still alive.

More Detail:

1. We can see this happening in the decoding block
   

   models/research/object_detection/data_decoders/tf_example_decoder.py

   Lines 285 to 293 in b9ef963

   	if num_keypoints > 0:
   	self.keys_to_features['image/object/keypoint/x'] = (
   	tf.VarLenFeature(tf.float32))
   	self.keys_to_features['image/object/keypoint/y'] = (
   	tf.VarLenFeature(tf.float32))
   	self.items_to_handlers[fields.InputDataFields.groundtruth_keypoints] = (
   	slim_example_decoder.ItemHandlerCallback(
   	['image/object/keypoint/y', 'image/object/keypoint/x'],
   	self._reshape_keypoints))

2. The mapping specifically happens at:
   

   models/research/object_detection/data_decoders/tf_example_decoder.py

   Lines 290 to 293 in b9ef963

   	self.items_to_handlers[fields.InputDataFields.groundtruth_keypoints] = (
   	slim_example_decoder.ItemHandlerCallback(
   	['image/object/keypoint/y', 'image/object/keypoint/x'],
   	self._reshape_keypoints))

   Where fields.InputDataFields.groundtruth_keypoints is defined here:
   

   models/research/object_detection/core/standard_fields.py

   Line 94 in b9ef963

   groundtruth_keypoints = 'groundtruth_keypoints'

3. Trying to use the higher level dataset_builder.build doesn't pass a num_keypoints and this defaults to 0.
   

   models/research/object_detection/builders/dataset_builder.py

   Lines 124 to 131 in b9ef963

   	decoder = tf_example_decoder.TfExampleDecoder(
   	load_instance_masks=input_reader_config.load_instance_masks,
   	load_multiclass_scores=input_reader_config.load_multiclass_scores,
   	instance_mask_type=input_reader_config.mask_type,
   	label_map_proto_file=label_map_proto_file,
   	use_display_name=input_reader_config.use_display_name,
   	num_additional_channels=input_reader_config.num_additional_channels)

   models/research/object_detection/data_decoders/tf_example_decoder.py

   Lines 136 to 148 in b9ef963

   	class TfExampleDecoder(data_decoder.DataDecoder):
   	"""Tensorflow Example proto decoder."""
   	def __init__(self,
   	load_instance_masks=False,
   	instance_mask_type=input_reader_pb2.NUMERICAL_MASKS,
   	label_map_proto_file=None,
   	use_display_name=False,
   	dct_method='',
   	num_keypoints=0,
   	num_additional_channels=0,
   	load_multiclass_scores=False):
   	"""Constructor sets keys_to_features and items_to_handlers.

[pharrellyhy]

Hi, @dustindorroh. Very useful information and I'm curious that have you successfully trained keypoints regression with object detection API? Thanks.

[pharrellyhy]

@dustindorroh I can create tfrecord with keypoints as labels and set num_keypoints=1. During training, the keypoint labels are loaded but I don't think the computation graph contains the keypoint output nodes. I can't find where these nodes are added to the graph automatically, so maybe we need to add them manually?

[soldierofhell]

Hi, is anyone working on keypoint functionality now? IMO there's still ~50% of work to do, basicly adding keypoint head to faster rcnn meta architecture. I can do this, but don't want to waste time if the work is in progress or sheduled?

[bhack]

Keypoints API is useful also for new anchor free models:
/cc @see--
https://github.com/see--/keras-centernet
https://github.com/princeton-vl/CornerNet-Lite

[bhack]

A new anchor free model https://github.com/Star-Clouds/CenterFace
We need to support bbox + keypoints API for anchorfree models.

[bhack]

See also tensorflow/hub#424

[bhack]

I added a Keypoints/Heatmaps issue in Tensorflow addons at tensorflow/addons#1366

[kensuke-ueda]

It seems that you should specify "KeypointBoxCoder" in pipeline.config to enable keypoint detection feature.

--------------------------------------- default pipeline.config
box_coder {
faster_rcnn_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}
--------------------------------------- you should change like this
box_coder {
keypoint_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}

[vesor]

I have been extending the box_coder to add keypoints detection for some time. I have tested on both Faster RCNN and SSD with customized training data which has keypoints labeling. The problem is it is a bit confusing with the keypoints concept in Mask RCNN related code. In Mask RCNN, keypoints are separated from box coding.

[pharrellyhy]

Hi @vesor , nice work! I think what your are saying is totally reasonable. If you don't mind, can you give some sample code on how to achieve that? Thanks!

[ghost]

@pharrellyhy @vesor could you share some information on training key points detection using TF object detection API?

[pharrellyhy]

@gsadhasivam It's easy if you are using TF2 and CenterNet.

[salpert-humane]

@dustindorroh - Are the keypoints expected to be normalized as the bounding boxes? (normalization by image width and height).

[dustindorroh]

That sounds correct. It’s been a while, but I remember keeping the keypoints normed with respect to the bounding box. The keypoint bookkeeping is a pain otherwise. From my own personal experience/mistakes I made, I had the bounding box coordinate transform backwards from the key point transform…. Or was it the other way… lol. Best of luck! Dustin

…

On Tue, Jun 29, 2021 at 11:46 AM salpert-humane ***@***.***> wrote: @dustindorroh <https://github.com/dustindorroh> - Are the keypoints expected to be normalized as the bounding boxes? (normalization by image width and height). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2366 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHWZ752D6XISX5I67BRJXTTVIIKBANCNFSM4D2NPFJA> .

[aidansmyth95]

Hi all,

Any advice on what the best way is today to train an SSD model with keypoints? The boxcoder seems the most promising answer. Can't use CenterNet unfortunately for compute resource reasons.

[dustindorroh]

Hello, The way I was able to accomplish this was to have two models. One SSD model for the bbox. The second model used the cropped part of the image to feed the landmarking model. I also built my landmarking model off of the SSD backbone. Best of luck! Dustin Dorroh

…

On Tue, Apr 18, 2023 at 6:29 PM aidansmyth95 ***@***.***> wrote: Hi all, Any advice on what the best way is today to train an SSD model with keypoints? The boxcoder seems the most promising answer. Can't use CenterNet unfortunately for compute resource reasons. — Reply to this email directly, view it on GitHub <#2366 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHWZ7322WQ3YVAXTMZY5Q3XB45YVANCNFSM4D2NPFJA> . You are receiving this because you were mentioned.Message ID: ***@***.***>