Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial Noise 是怎么统计得来 #12

Open
13718413797 opened this issue Jan 17, 2025 · 1 comment
Open

Initial Noise 是怎么统计得来 #12

13718413797 opened this issue Jan 17, 2025 · 1 comment

Comments

@13718413797
Copy link

Initial Noise 是怎么统计得来的 是否有相关的代码demo

@zhuhz22
Copy link
Collaborator

zhuhz22 commented Jan 17, 2025

Hi @13718413797 ,
Thank you for your attention to our work! We traverse the video, passing them through the VAE to encode, then compute the mean and variance of these latents for the initial noise.

An illustrative demo is as follows (for simplicity, the specific function details are omitted) :

    cnt=0
    with torch.no_grad(), torch.cuda.amp.autocast():
        # ----------- Traverse the video dataset to obtain its latent space tensor. ---------------
        for root, dirs, files in os.walk(gt_path):
            for file in files[:args.total-1]:
                file_path = os.path.join(root, file)
                gt_video=get_gt(file_path,video_size=(args.height, args.width)).to("cuda")
                gt_video=get_latent_z(model,gt_video)
                # ---------- For the mean, simply sum each tensor element-wise and divide by the total number. -----
                # For variance, we use the formula Var(x) = E(x²) - E²(x). Therefore, we need to calculate E(x²), which is computed as gt_video**2 / sum.
                if cnt==0:
                    sum=gt_video
                    sum_for_var=gt_video**2
                else:
                    sum+=gt_video
                    sum_for_var+=gt_video**2
                cnt+=1
                
        # The mean has been calculated.
        expectation_X_0=sum/cnt
        # Here, the sum of the element-wise variances is calculated.
        tr_covar_X_0=sum_for_var/cnt-expectation_X_0**2
        tr_Cov=tr_covar_X_0.sum()
        # Divide the sum of variances by the total number of elements in the tensor.
        shape_ = expectation_X_0.shape
        d = 1
        for dim in shape_:
            d *= dim
        var=tr_Cov/d
        #------------------After calculating the mean and variance, apply the final weighting.-------------
    sqrt_alpha_t=model.get_sqrt_alpha_t_bar(gt_video,torch.tensor([args.sdedit_t-1]).to('cuda'))
    # Final mean.
    mu_p=sqrt_alpha_t*expectation_X_0
    # Final std
    alpha_t=sqrt_alpha_t**2
    sigma_p=torch.sqrt(1-alpha_t + alpha_t*var) 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants