Skip to content

Commit

Permalink
Adding more fitting python bindings (#354)
Browse files Browse the repository at this point in the history 
* Added a `fit_shape_and_pose()` overload that takes in a LandmarkCollection and LandmarkMapper. The new overloads map the landmarks, and then call the existing `fit_shape_and_pose()` function. The Blender plugin uses this functionality.
* Added `MorphableModel::set_landmark_definitions()` and improved MorphableModel docs slightly

---------

Co-authored-by: Patrik Huber <[email protected]>
  • Loading branch information
@omar-h-omar @patrikhuber
omar-h-omar and patrikhuber authored Aug 26, 2023
1 parent 72aff97 commit fbb0763
Show file tree
Hide file tree
Showing 3 changed files with 194 additions and 5 deletions.
149 changes: 149 additions & 0 deletions include/eos/fitting/fitting.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -778,6 +778,155 @@ inline std::pair<core::Mesh, fitting::RenderingParameters> fit_shape_and_pose(
blendshape_coeffs, fitted_image_points);
};

/**
* @brief Fit the pose (camera), shape model, and expression blendshapes to landmarks,
* in an iterative way.
*
* Convenience function that fits pose (camera), the shape model, and expression blendshapes
* to landmarks, in an iterative (alternating) way. It uses the given, fixed landmarks-to-vertex
* correspondences, and does not use any dynamic contour fitting.
*
* If \p pca_shape_coefficients and/or \p blendshape_coefficients are given, they are used as
* starting values in the fitting. When the function returns, they contain the coefficients from
* the last iteration.
*
* \p num_iterations: Results are good for even a single iteration. For single-image fitting and
* for full convergence of all parameters, it can take up to 300 iterations. In tracking,
* particularly if initialising with the previous frame, it works well with as low as 1 to 5
* iterations.
* \p edge_topology is used for the occluding-edge face contour fitting.
* \p contour_landmarks and \p model_contour are used to fit the front-facing contour.
*
* Note: If the given \p morphable_model contains a PCA expression model, alternating the shape identity and
* expression fitting is theoretically not needed - the basis matrices could be stacked, and then both
* coefficients could be solved for in one go. The two bases are most likely not orthogonal though.
* In any case, alternating hopefully doesn't do any harm.
*
* Todo: Add a convergence criterion.
*
* @param[in] morphable_model The 3D Morphable Model used for the shape fitting.
* @param[in] landmarks A LandmarkCollection of 2D landmarks.
* @param[in] landmark_mapper A mapper which maps the 2D landmark identifiers to 3D model vertex indices.
* @param[in] image_width Width of the input image (needed for the camera model).
* @param[in] image_height Height of the input image (needed for the camera model).
* @param[in] num_iterations Number of iterations that the different fitting parts will be alternated for.
* @param[in] num_shape_coefficients_to_fit How many shape-coefficients to fit (all others will stay 0). Should be bigger than zero, or std::nullopt to fit all coefficients.
* @param[in] lambda_identity Regularisation parameter of the PCA shape fitting.
* @param[in] num_expression_coefficients_to_fit How many shape-coefficients to fit (all others will stay 0). Should be bigger than zero, or std::nullopt to fit all coefficients. Only used for expression-PCA fitting.
* @param[in] lambda_expressions Regularisation parameter of the expression fitting. Only used for expression-PCA fitting.
* @param[in,out] pca_shape_coefficients If given, will be used as initial PCA shape coefficients to start the fitting. Will contain the final estimated coefficients.
* @param[in,out] expression_coefficients If given, will be used as initial expression blendshape coefficients to start the fitting. Will contain the final estimated coefficients.
* @param[out] fitted_image_points Debug parameter: Returns all the 2D points that have been used for the fitting.
* @return The fitted model shape instance and the final pose.
*/
inline std::pair<core::Mesh, fitting::RenderingParameters> fit_shape_and_pose(
const morphablemodel::MorphableModel& morphable_model,
const core::LandmarkCollection<Eigen::Vector2f>& landmarks, const core::LandmarkMapper& landmark_mapper,
int image_width, int image_height, int num_iterations,
cpp17::optional<int> num_shape_coefficients_to_fit, float lambda_identity,
cpp17::optional<int> num_expression_coefficients_to_fit, cpp17::optional<float> lambda_expressions,
std::vector<float>& pca_shape_coefficients, std::vector<float>& expression_coefficients,
std::vector<Eigen::Vector2f>& fitted_image_points)
{
assert(landmarks.size() >= 4);
assert(image_width > 0 && image_height > 0);
assert(num_iterations > 0); // Can we allow 0, for only the initial pose-fit?
assert(pca_shape_coefficients.size() <= morphable_model.get_shape_model().get_num_principal_components());
// More asserts I forgot?
if (num_shape_coefficients_to_fit)
{
if (num_shape_coefficients_to_fit.value() >
morphable_model.get_shape_model().get_num_principal_components())
{
throw std::runtime_error(
"Specified more shape coefficients to fit than the given shape model contains.");
}
}

using Eigen::MatrixXf;
using Eigen::Vector2f;
using Eigen::Vector4f;
using Eigen::VectorXf;
using std::vector;

// The 2D and 3D point correspondences used for the fitting:
vector<int> vertex_indices; // their vertex indices
vector<Vector2f> image_points; // the corresponding 2D landmark points

// Sub-select all the landmarks which we have a mapping for (i.e. that are defined in the 3DMM),
// and get the corresponding model points (mean if given no initial coeffs, from the computed shape otherwise):
for (int i = 0; i < landmarks.size(); ++i)
{
const auto converted_name = landmark_mapper.convert(landmarks[i].name);
if (!converted_name)
{ // no mapping defined for the current landmark
continue;
}
// If the MorphableModel does not contain landmark definitions, we expect the user to have given us
// direct mappings (e.g. directly from ibug identifiers to vertex ids). If the model does contain
// landmark definitions, we expect the user to use mappings from their landmark identifiers (e.g.
// ibug) to the landmark definitions, and not to vertex indices.
// Todo: This might be worth mentioning in the function documentation of fit_shape_and_pose.
int vertex_idx;
if (morphable_model.get_landmark_definitions())
{
const auto found_vertex_idx =
morphable_model.get_landmark_definitions().value().find(converted_name.value());
if (found_vertex_idx != std::end(morphable_model.get_landmark_definitions().value()))
{
vertex_idx = found_vertex_idx->second;
} else
{
continue;
}
} else
{
vertex_idx = std::stoi(converted_name.value());
}
// model_points.emplace_back(current_mesh.vertices[vertex_idx].homogeneous());
vertex_indices.emplace_back(vertex_idx);
image_points.emplace_back(landmarks[i].coordinates);
}

return fit_shape_and_pose(morphable_model, image_points, vertex_indices, image_width, image_height,
num_iterations, num_shape_coefficients_to_fit, lambda_identity,
num_expression_coefficients_to_fit, lambda_expressions, pca_shape_coefficients,
expression_coefficients, fitted_image_points);
};

/**
* @brief Fit the pose (camera), shape model, and expression blendshapes to landmarks,
* in an iterative way, given fixed landmarks-to-vertex correspondences.
*
* This is a convenience overload without the last 3 in/out parameters of above function.
*
* @param[in] morphable_model The 3D Morphable Model used for the shape fitting.
* @param[in] landmarks A LandmarkCollection of 2D landmarks.
* @param[in] landmark_mapper A mapper which maps the 2D landmark identifiers to 3D model vertex indices.
* @param[in] image_width Width of the input image (needed for the camera model).
* @param[in] image_height Height of the input image (needed for the camera model).
* @param[in] num_iterations Number of iterations that the different fitting parts will be alternated for.
* @param[in] num_shape_coefficients_to_fit How many shape-coefficients to fit (all others will stay 0). Should be bigger than zero, or std::nullopt to fit all coefficients.
* @param[in] lambda_identity Regularisation parameter of the PCA shape fitting.
* @param[in] num_expression_coefficients_to_fit How many shape-coefficients to fit (all others will stay 0). Should be bigger than zero, or std::nullopt to fit all coefficients. Only used for expression-PCA fitting.
* @param[in] lambda_expressions Regularisation parameter of the expression fitting. Only used for expression-PCA fitting.
* @return The fitted model shape instance and the final pose.
*/
inline std::pair<core::Mesh, fitting::RenderingParameters> fit_shape_and_pose(
const morphablemodel::MorphableModel& morphable_model, const core::LandmarkCollection<Eigen::Vector2f>& landmarks,
const core::LandmarkMapper& landmark_mapper, int image_width, int image_height, int num_iterations,
cpp17::optional<int> num_shape_coefficients_to_fit, float lambda_identity,
cpp17::optional<int> num_expression_coefficients_to_fit, cpp17::optional<float> lambda_expressions)
{
std::vector<float> pca_coeffs;
std::vector<float> blendshape_coeffs;
std::vector<Eigen::Vector2f> fitted_image_points;
return fit_shape_and_pose(morphable_model, landmarks, landmark_mapper, image_width, image_height,
num_iterations, num_shape_coefficients_to_fit, lambda_identity,
num_expression_coefficients_to_fit, lambda_expressions, pca_coeffs,
blendshape_coeffs, fitted_image_points);
};

} /* namespace fitting */
} /* namespace eos */

Expand Down
22 changes: 17 additions & 5 deletions include/eos/morphablemodel/MorphableModel.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,12 @@ core::Mesh sample_to_mesh(
const std::vector<std::array<int, 3>>& texture_triangle_indices = std::vector<std::array<int, 3>>());

/**
* @brief A class representing a 3D Morphable Model, consisting
* of a shape- and colour (albedo) PCA model.
* @brief A class representing a 3D Morphable Model, consisting of a shape- and colour (albedo) PCA model. It
* can additionally contain an optional expression model, which can be a PCA model or consist of linear
* expression blendshapes.
*
* For the general idea of 3DMMs see T. Vetter, V. Blanz,
* 'A Morphable Model for the Synthesis of 3D Faces', SIGGRAPH 1999.
* For the general idea of 3DMMs see T. Vetter, V. Blanz, 'A Morphable Model for the Synthesis of 3D Faces',
* SIGGRAPH 1999.
*/
class MorphableModel
{
Expand Down Expand Up @@ -394,7 +395,7 @@ class MorphableModel
* Returns the landmark definitions for this Morphable Model, which might be an empty optional, if the
* model doesn't contain any.
*
* The landmark definitions are define mappings from a set of global landmark identifiers, like for
* The landmark definitions define mappings from a set of global landmark identifiers, like for
* example "eye.right.outer_corner", to the model's respective vertex indices.
* A MorphableModel may or may not contain these landmark definitions, depending on how it was created.
*
Expand All @@ -405,6 +406,17 @@ class MorphableModel
return landmark_definitions;
};

/**
* Sets the landmark definitions for this Morphable Model.
*
* The landmark definitions define mappings from a set of global landmark identifiers, like for
* example "eye.right.outer_corner", to the model's respective vertex indices.
*/
void set_landmark_definitions(cpp17::optional<std::unordered_map<std::string, int>> landmark_definitions)
{
this->landmark_definitions = landmark_definitions;
};

/**
* Returns the texture coordinates for all the vertices in the model.
*
Expand Down
28 changes: 28 additions & 0 deletions python/generate-python-bindings.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,7 @@ PYBIND11_MODULE(eos, eos_module)
.def("has_color_model", &morphablemodel::MorphableModel::has_color_model, "Returns true if this Morphable Model contains a colour model, and false if it is a shape-only model.")
.def("has_separate_expression_model", &morphablemodel::MorphableModel::has_separate_expression_model, "Returns true if this Morphable Model contains a separate PCA or Blendshapes expression model.")
.def("get_landmark_definitions", &morphablemodel::MorphableModel::get_landmark_definitions, "Returns the landmark definitions for this Morphable Model, which might be an empty optional, if the model doesn't contain any.")
.def("set_landmark_definitions", &morphablemodel::MorphableModel::set_landmark_definitions, "Sets the landmark definitions for this Morphable Model.", py::arg("landmark_definitions"))
.def("get_texture_coordinates", &morphablemodel::MorphableModel::get_texture_coordinates, "Returns the texture coordinates for all the vertices in the model.")
.def("get_texture_triangle_indices", &morphablemodel::MorphableModel::get_texture_triangle_indices, "Returns the triangulation (the triangles that make up the uv mapping) for the texture coordinates.")
.def("get_expression_model_type", &morphablemodel::MorphableModel::get_expression_model_type, "Returns the type of the expression model: None, Blendshapes or PcaModel.");
Expand Down Expand Up @@ -337,6 +338,33 @@ PYBIND11_MODULE(eos, eos_module)
py::arg("num_shape_coefficients_to_fit") = py::none(), py::arg("lambda_identity") = 30.0f,
py::arg("num_expression_coefficients_to_fit") = py::none(), py::arg("lambda_expressions") = 30.0f);

fitting_module.def(
"fit_shape_and_pose",
[](const morphablemodel::MorphableModel& morphable_model,
const core::LandmarkCollection<Eigen::Vector2f>& landmarks,
const core::LandmarkMapper& landmark_mapper, int image_width, int image_height,
int num_iterations, cpp17::optional<int> num_shape_coefficients_to_fit, float lambda_identity,
cpp17::optional<int> num_expression_coefficients_to_fit,
cpp17::optional<float> lambda_expressions,
std::vector<float> pca_coeffs,
std::vector<float> blendshape_coeffs,
std::vector<Eigen::Vector2f> fitted_image_points) {
const auto result = fitting::fit_shape_and_pose(
morphable_model, landmarks, landmark_mapper, image_width, image_height,
num_iterations, num_shape_coefficients_to_fit,
lambda_identity, num_expression_coefficients_to_fit, lambda_expressions, pca_coeffs,
blendshape_coeffs, fitted_image_points);
return std::make_tuple(result.first, result.second, pca_coeffs, blendshape_coeffs);
},
"Fit the pose (camera), shape model, and expression blendshapes to landmarks, in an iterative way. "
"Returns a tuple (mesh, rendering_parameters, shape_coefficients, blendshape_coefficients).",
py::arg("morphable_model"), py::arg("landmarks"), py::arg("landmark_mapper"), py::arg("image_width"),
py::arg("image_height"), py::arg("num_iterations") = 5,
py::arg("num_shape_coefficients_to_fit") = py::none(), py::arg("lambda_identity") = 30.0f,
py::arg("num_expression_coefficients_to_fit") = py::none(), py::arg("lambda_expressions") = 30.0f,
py::arg("pca_coeffs") = std::vector<float>(), py::arg("blendshape_coeffs") = std::vector<float>(),
py::arg("fitted_image_points") = std::vector<Eigen::Vector2f>());

fitting_module.def(
"fit_shape_to_landmarks_linear",
[](const morphablemodel::PcaModel& shape_model,
Expand Down

0 comments on commit fbb0763

Please sign in to comment.