How to create "_motion.flo" file for target image

[shangdibufashi]

This is a really amazing project.
In the "sample_CLAWv2.zip", images (e.g. 00139.png ) has its own ".flo" file(e.g. 00139_motion.flo)
Wondering how to create "_motion.flo" file when testing my own images
Thanks

[simon3dv]

Hi @shangdibufashi , we provide a script to inference motion with sparse hints and mask described in README using the model "baseline2+motion2+fixedMotionFinetune".
bash test_animating/CLAW/test_baseline_motion.sh
In the script, mask is generated from ground-truth motion using a threshold , and 5 sparse hints are sampled from ground truth motion using K-Means:

SLR-SFS/test_animating/test_motion_4eval_rawsize_threshold.py

Lines 174 to 192 in 8d922b3

	big_motion_alpha = (gt_motion_speed > 0.2161635).float()
	#moving_region_mask = big_motion_alpha
	if int(big_motion_alpha.sum().long()) < 5:
	dense_motion = torch.zeros(gt_motion.shape)
	else:
	max_hint = 5
	np.random.seed(5)
	estimator = KMeans(n_clusters=max_hint)
	# index = torch.randint(0, int(big_motion_alpha.sum()), (max_hint,))
	hint_y = torch.zeros((max_hint,))
	hint_x = torch.zeros((max_hint,))
	big_motion_xys = xys[:, torch.where(big_motion_alpha.view(1, 1, height * width))[2]] # 2, M
	X = big_motion_xys.permute(1, 0).cpu().detach().numpy()
	estimator.fit(X)
	labels = estimator.labels_
	for i in range(max_hint):
	selected_xy = X[labels == i].mean(0)
	hint_y[i] = int(selected_xy[1])
	hint_x[i] = int(selected_xy[0])

For your own image, you need to modify the value of hint_y, hint_x(motion speed of 1~5 pixels), big_motion_alpha(mask with motion).
Sorry that the UI is not provided yet. Maybe you can refer to Labelme to generate the mask, and directly edit hint value in the code to have a quick try.

[shangdibufashi]

great, thanks for your detailed guidance:)

[FYtrace]

Hi @shangdibufashi , we provide a script to inference motion with sparse hints and mask described in README using the model "baseline2+motion2+fixedMotionFinetune". bash test_animating/CLAW/test_baseline_motion.sh In the script, mask is generated from ground-truth motion using a threshold , and 5 sparse hints are sampled from ground truth motion using K-Means:

SLR-SFS/test_animating/test_motion_4eval_rawsize_threshold.py

Lines 174 to 192 in 8d922b3

	big_motion_alpha = (gt_motion_speed > 0.2161635).float()
	#moving_region_mask = big_motion_alpha
	if int(big_motion_alpha.sum().long()) < 5:
	dense_motion = torch.zeros(gt_motion.shape)
	else:
	max_hint = 5
	np.random.seed(5)
	estimator = KMeans(n_clusters=max_hint)
	# index = torch.randint(0, int(big_motion_alpha.sum()), (max_hint,))
	hint_y = torch.zeros((max_hint,))
	hint_x = torch.zeros((max_hint,))
	big_motion_xys = xys[:, torch.where(big_motion_alpha.view(1, 1, height * width))[2]] # 2, M
	X = big_motion_xys.permute(1, 0).cpu().detach().numpy()
	estimator.fit(X)
	labels = estimator.labels_
	for i in range(max_hint):
	selected_xy = X[labels == i].mean(0)
	hint_y[i] = int(selected_xy[1])
	hint_x[i] = int(selected_xy[0])

For your own image, you need to modify the value of hint_y, hint_x(motion speed of 1~5 pixels), big_motion_alpha(mask with motion).
Sorry that the UI is not provided yet. Maybe you can refer to Labelme to generate the mask, and directly edit hint value in the code to have a quick try.

Thanks for your excellent work. I got some trouble re-producing Initial Motion from Interactive Sparse Labeling, It seems that we should not only get the coordinates of 1~5 pixels but also need to get motion direction and speed. Can you share the code that produces the sparse flow from the sampled point or interactive sampling map?

[abhi970715]

@shangdibufashi can you help me in running this project please

[chouchonkit]

Hi @shangdibufashi , we provide a script to inference motion with sparse hints and mask described in README using the model "baseline2+motion2+fixedMotionFinetune". bash test_animating/CLAW/test_baseline_motion.sh In the script, mask is generated from ground-truth motion using a threshold , and 5 sparse hints are sampled from ground truth motion using K-Means:

SLR-SFS/test_animating/test_motion_4eval_rawsize_threshold.py

Lines 174 to 192 in 8d922b3

	big_motion_alpha = (gt_motion_speed > 0.2161635).float()
	#moving_region_mask = big_motion_alpha
	if int(big_motion_alpha.sum().long()) < 5:
	dense_motion = torch.zeros(gt_motion.shape)
	else:
	max_hint = 5
	np.random.seed(5)
	estimator = KMeans(n_clusters=max_hint)
	# index = torch.randint(0, int(big_motion_alpha.sum()), (max_hint,))
	hint_y = torch.zeros((max_hint,))
	hint_x = torch.zeros((max_hint,))
	big_motion_xys = xys[:, torch.where(big_motion_alpha.view(1, 1, height * width))[2]] # 2, M
	X = big_motion_xys.permute(1, 0).cpu().detach().numpy()
	estimator.fit(X)
	labels = estimator.labels_
	for i in range(max_hint):
	selected_xy = X[labels == i].mean(0)
	hint_y[i] = int(selected_xy[1])
	hint_x[i] = int(selected_xy[0])

For your own image, you need to modify the value of hint_y, hint_x(motion speed of 1~5 pixels), big_motion_alpha(mask with motion).
Sorry that the UI is not provided yet. Maybe you can refer to Labelme to generate the mask, and directly edit hint value in the code to have a quick try.

Thanks for your excellent work. I got some trouble re-producing Initial Motion from Interactive Sparse Labeling, It seems that we should not only get the coordinates of 1~5 pixels but also need to get motion direction and speed. Can you share the code that produces the sparse flow from the sampled point or interactive sampling map?

Do you solve this problem?

[simon3dv]

https://github.com/simon3dv/SLR-SFS/blob/8d922b32c895aa6d53e71241318bf0f677a6d6be/test_animating/test_motion_4eval_rawsize_threshold.py#LL162C1-L209C89

Replace to
to (for example)

                max_hint = 1
                hint_y = torch.zeros((max_hint,))
                hint_x = torch.zeros((max_hint,))
                hint_y[0] = height // 2 # posision
                hint_x[0] = width // 2 # posision
                speed = torch.zeros((1,2, max_hint))
                speed[0,0,0] = 1 # Speed of x direction
                speed[0,1,0] = 2 # Speed of y direction
                for i_hint in range(max_hint):
                    dist = ((xys - xys.view(2, height, width)[:, hint_y[i_hint], hint_x[i_hint]].unsqueeze(
                        1)) ** 2).sum(0, True).sqrt()  # 1,W*W
                    weight = (-(dist / sigma) ** 2).exp().unsqueeze(0)
                    dense_motion += weight * speed[...,i_hint].unsqueeze(2)
                    dense_motion_norm += weight

The hint_x, hint_y could be get from start point of an arrow, and speed can be get from length of an arrow. (Similar to Motion Leap app https://apps.apple.com/us/app/enlight-pixaloop/id1381206010)
I haven't not develop an interative UI to edit and render instantly for the time being, I just use sensebee to label mask and arrow and translate to these values.

[x4jc1]

Hi @shangdibufashi , we provide a script to inference motion with sparse hints and mask described in README using the model "baseline2+motion2+fixedMotionFinetune". bash test_animating/CLAW/test_baseline_motion.sh In the script, mask is generated from ground-truth motion using a threshold , and 5 sparse hints are sampled from ground truth motion using K-Means:

SLR-SFS/test_animating/test_motion_4eval_rawsize_threshold.py

Lines 174 to 192 in 8d922b3

	big_motion_alpha = (gt_motion_speed > 0.2161635).float()
	#moving_region_mask = big_motion_alpha
	if int(big_motion_alpha.sum().long()) < 5:
	dense_motion = torch.zeros(gt_motion.shape)
	else:
	max_hint = 5
	np.random.seed(5)
	estimator = KMeans(n_clusters=max_hint)
	# index = torch.randint(0, int(big_motion_alpha.sum()), (max_hint,))
	hint_y = torch.zeros((max_hint,))
	hint_x = torch.zeros((max_hint,))
	big_motion_xys = xys[:, torch.where(big_motion_alpha.view(1, 1, height * width))[2]] # 2, M
	X = big_motion_xys.permute(1, 0).cpu().detach().numpy()
	estimator.fit(X)
	labels = estimator.labels_
	for i in range(max_hint):
	selected_xy = X[labels == i].mean(0)
	hint_y[i] = int(selected_xy[1])
	hint_x[i] = int(selected_xy[0])

For your own image, you need to modify the value of hint_y, hint_x(motion speed of 1~5 pixels), big_motion_alpha(mask with motion).
Sorry that the UI is not provided yet. Maybe you can refer to Labelme to generate the mask, and directly edit hint value in the code to have a quick try.

great work! may i ask you how to transfer a mask image (a png file) to the big_motion_alpha?

[x4jc1]

Hi @shangdibufashi , we provide a script to inference motion with sparse hints and mask described in README using the model "baseline2+motion2+fixedMotionFinetune". bash test_animating/CLAW/test_baseline_motion.sh In the script, mask is generated from ground-truth motion using a threshold , and 5 sparse hints are sampled from ground truth motion using K-Means:

SLR-SFS/test_animating/test_motion_4eval_rawsize_threshold.py

Lines 174 to 192 in 8d922b3

	big_motion_alpha = (gt_motion_speed > 0.2161635).float()
	#moving_region_mask = big_motion_alpha
	if int(big_motion_alpha.sum().long()) < 5:
	dense_motion = torch.zeros(gt_motion.shape)
	else:
	max_hint = 5
	np.random.seed(5)
	estimator = KMeans(n_clusters=max_hint)
	# index = torch.randint(0, int(big_motion_alpha.sum()), (max_hint,))
	hint_y = torch.zeros((max_hint,))
	hint_x = torch.zeros((max_hint,))
	big_motion_xys = xys[:, torch.where(big_motion_alpha.view(1, 1, height * width))[2]] # 2, M
	X = big_motion_xys.permute(1, 0).cpu().detach().numpy()
	estimator.fit(X)
	labels = estimator.labels_
	for i in range(max_hint):
	selected_xy = X[labels == i].mean(0)
	hint_y[i] = int(selected_xy[1])
	hint_x[i] = int(selected_xy[0])

For your own image, you need to modify the value of hint_y, hint_x(motion speed of 1~5 pixels), big_motion_alpha(mask with motion).
Sorry that the UI is not provided yet. Maybe you can refer to Labelme to generate the mask, and directly edit hint value in the code to have a quick try.

great work! may i ask you how to transfer a mask image (a png file) to the big_motion_alpha?

@simon3dv

[simon3dv]

@x4jc1 big_motion_alpha is the binary mask of fluid which you want to animate, so just load the png file to big_motion_alpha and convert it to float datatype.

[Spark-RTG]

Hello! Has this been solved? I hand-drew a mask for the fluid in the image, but I got this error after getting the big_motion_alpha correct in shape as (1,1, height, width). After tweaking with the code for over 6 hours I think I should reach out here. (I am trying to animate a scene created by Midjourney (AI-drawing)

Traceback (most recent call last):
  File "/content/SLR-SFS/test_animating/test_motion_4eval_rawsize_threshold.py", line 209, in <module>
    flow = model_to_test.model.module.motion_regressor.forward_flow(img, big_motion_alpha.cuda(), hint.cuda())["PredMotion"]
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'AnimatingSoftmaxSplatingJoint' object has no attribute 'motion_regressor'

It seems to be caused by the incomplete pretrained model. I have tried both the
"baseline2+motion2+fixedMotionFinetune" model and the "v1_project_page_bestloss_ep100+50" model, and neither of those helped (the "/mnt/lustre/fansiming/project/.../model_epoch.pthbest100" serving as default model could also not be found in the directory). Could you help check why this is happening?

FYI, here is my modified "test_motion_4eval_rawsize_threshold.py" file (github doesn't support .py so I used .txt) Thank you so much for maintaining this excellent project!

test_motion_4eval_rawsize_threshold.txt

Here is my picture and mask. I would greatly appreciate it if you wish to run them as example tests.

[simon3dv]

Hi @Spark-RTG , "AttributeError: 'AnimatingSoftmaxSplatingJoint' object has no attribute 'motion_regressor'", the motion_regressor should be in this pretrained model: baseline2+motion2+fixedMotionFinetune.pth

import torch
model = torch.load("baseline2+motion2+fixedMotionFinetune.pth")
print([x for x in model['state_dict'].keys() if 'motion_regressor' in x])

Sorry that I am working on other projects and have no time running the image. But I guess the animation result might not be good since (a)the motion regressor is trained in real scenes with good lighting, (b) It has no transparent pattern, and (c) the river looks static in global, but containing randomly small local waves. The solution:
(a) Since our final result is very similar to the Motion Leap app.(We both use euler Integration to generate video motions from single motion). I suggest you try this app first if you have enough time to edit the one single image.
(b) If the alpha prediction is strange, switch the v2 model to v1(single layer model),
(c) If you want to generate randomly waves motions, find a similar video(such as from storyblocks), use flownet2 to predict the video motion of each frame pair. (1->2, 2->3, ... 29->30), then replace these motion to the ones from euler integration.

[AIFFFENG]

Hi @shangdibufashi , we provide a script to inference motion with sparse hints and mask described in README using the model "baseline2+motion2+fixedMotionFinetune". bash test_animating/CLAW/test_baseline_motion.sh In the script, mask is generated from ground-truth motion using a threshold , and 5 sparse hints are sampled from ground truth motion using K-Means:

SLR-SFS/test_animating/test_motion_4eval_rawsize_threshold.py

Lines 174 to 192 in 8d922b3

	big_motion_alpha = (gt_motion_speed > 0.2161635).float()
	#moving_region_mask = big_motion_alpha
	if int(big_motion_alpha.sum().long()) < 5:
	dense_motion = torch.zeros(gt_motion.shape)
	else:
	max_hint = 5
	np.random.seed(5)
	estimator = KMeans(n_clusters=max_hint)
	# index = torch.randint(0, int(big_motion_alpha.sum()), (max_hint,))
	hint_y = torch.zeros((max_hint,))
	hint_x = torch.zeros((max_hint,))
	big_motion_xys = xys[:, torch.where(big_motion_alpha.view(1, 1, height * width))[2]] # 2, M
	X = big_motion_xys.permute(1, 0).cpu().detach().numpy()
	estimator.fit(X)
	labels = estimator.labels_
	for i in range(max_hint):
	selected_xy = X[labels == i].mean(0)
	hint_y[i] = int(selected_xy[1])
	hint_x[i] = int(selected_xy[0])

For your own image, you need to modify the value of hint_y, hint_x(motion speed of 1~5 pixels), big_motion_alpha(mask with motion).
Sorry that the UI is not provided yet. Maybe you can refer to Labelme to generate the mask, and directly edit hint value in the code to have a quick try.

Thanks for your work, how can I generate the .flo file, directly through flownet2, knowing the entire video?

[lyyl08]

Hello, when I “Ours_v1_ProjectPaper‘’ through your model, I found that there is a .flo file in the operation required, I want to know how to get this .flo, and in this command "bash test_animating/CLAW/test_baseline_motion.sh" , the script still needs the input jpg file related to the flo file, but I don't know how to get the flo file, Can you answer that for me?How do I get this kind of FLO file?