Chapter03_ScipyOptimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8e8a8292-4f27-4fe8-8e39-20861146680f",
   "metadata": {},
   "source": [
    "<h1>Optimization Package</h1><br>\n",
    "<img src=\"https://img.icons8.com/external-itim2101-lineal-color-itim2101/344/external-professor-life-style-avatar-itim2101-lineal-color-itim2101.png\" alt=\"Instructor\" width=50> In this section we check some of higer level tools for running gradient based optimization. We also, will talk about more advanced version of gradient descent methods and some theoritical ideas behind each version."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42b772fe-5a64-4594-a4ee-9a3caeb58da4",
   "metadata": {},
   "source": [
    "<img src=\"https://docs.scipy.org/doc/scipy/_static/logo.svg\" width=80> <b><font size=20>SciPy</font></b><br>\n",
    "Scipy is a package for scientific calculations. It contains modules for optimization, linear algebra, integration, interpolation, special functions, FFT, signal and image processing, ODE solvers and other tasks common in science and engineering. Scipy uses numpy as a backbone. Some functionalities are overlapped but many of them are just available in scipy.<br>\n",
    "For installing Scipy use: ```conda install -c anaconda scipy```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcbacaca-aac3-4787-9873-58984583b143",
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy\n",
    "import numpy as np # loding np for passing numbers to scipy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "032e2795-e3cf-458b-85d5-1325e7d52c49",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's do some examples before jumping to optimization\n",
    "# Scipy provide a great implementation of linear algebra\n",
    "from scipy import linalg\n",
    "a = np.array([[3, 2, 0], [1, -1, 0], [0, 5, 1]])\n",
    "b = np.array([2, 4, -1])\n",
    "linalg.inv(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24e7f550-ca3c-4821-bff9-506edd3564dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "linalg.solve(a,b) # The solution to linear system aX=b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fa80260-74e1-4782-8b04-b3c6ada3d4c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determinant\n",
    "linalg.det(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9194ff3-3951-4e61-91d8-c2d10cdb2aa4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Eigen decomposition\n",
    "linalg.eig(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16c89e4f-9a22-4200-a77c-16391e0ea31a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# singular Value Decomposition SVD\n",
    "U, s, Vh = linalg.svd(a)\n",
    "display(U,  s,  Vh)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5b01c28-adfa-40bd-a45a-506ecc1b2789",
   "metadata": {},
   "source": [
    "<img src=\"https://img.icons8.com/color/344/light.png\" width=70> There is a tone of useful and advanced linear algebra functions inside linalg. Also, Scipy.stats has very useful statistical tools. We just reviewed the tip of iceberg. When you have time, go and check documents."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65806b9d-3512-44fb-bc35-3438cb0da05f",
   "metadata": {},
   "source": [
    "<img src=\"https://img.icons8.com/external-itim2101-lineal-color-itim2101/344/external-professor-life-style-avatar-itim2101-lineal-color-itim2101.png\" alt=\"Instructor\" width=50> Scipy.optimize has optimization functions. There are few functions but support a variety of methods. Different methods are selected by providing an argument to the function. Let's review them. Also, there are specific methods for minimization of single variable functions. Let's review them first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efbe6de1-5277-4b98-947e-400c58909e83",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import optimize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dd332d4-2b14-46b7-8690-14194ce9cd88",
   "metadata": {},
   "source": [
    "<img src=\"https://img.icons8.com/external-itim2101-lineal-color-itim2101/344/external-professor-life-style-avatar-itim2101-lineal-color-itim2101.png\" alt=\"Instructor\" width=50>. For single variable function use `optimize.minimize_scalar`. It is based on <a href=\"https://en.wikipedia.org/wiki/Brent%27s_method\"> Brent</a> method. This method uses a bi-section search for finding the optimal point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28e279fa-09fe-4e1b-84de-8fd8997519de",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy import special # This module has many special function in math like Bessel, Hankel, Gamma, ...\n",
    "\n",
    "objective = special.j0 # This is the Bessel 0 function - Check here https://en.wikipedia.org/wiki/Bessel_function\n",
    "\n",
    "x = np.linspace(0,15, 1000)\n",
    "y = objective(x)\n",
    "plt.plot(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9637ec30-3efe-4a28-a529-4ee96a6860f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_opt = optimize.minimize_scalar(objective)\n",
    "x_opt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e8c945d-39ab-4043-a38c-e1a56ea7bc67",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(x_opt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "810b9cc2-a262-4232-be44-e81b8c2b2ef3",
   "metadata": {},
   "source": [
    "<img src=\"https://img.icons8.com/color/344/light.png\" width=50> <b>OptimizationResult:</b> <br>\n",
    "It keeps the outcome of the optimization. Depending on the meethod it can have more or less field. But, it generally contains:<br>\n",
    "<ul>\n",
    "    <li> <b>x</b>: The optimal point. </li>\n",
    "    <li><b>fun</b>: The value of objective at optimal point.</li>\n",
    "    <li><b>Success</b>: Whether the optimization problem converged to an answer or not</li>\n",
    "    <li><b> nit</b>: Number of iterations.</li>\n",
    "    <li><b> nfev</b>: Number of time the objective function is evaluated.</li>\n",
    "    </ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12297ac3-d501-40ac-a782-5519bc9ca6ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimize.minimize_scalar(objective, method=\"Golden\") # https://en.wikipedia.org/wiki/Golden-section_search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "849e8a77-a82b-4325-8393-3c3cf332d5e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimize.minimize_scalar(objective, method=\"bounded\", bounds=(8, 12)) # bounded implementation of Brent’s algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f22a8493-8a03-431c-ada5-0cda3bf30872",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Other methods have bracket.\n",
    "optimize.minimize_scalar(objective, method=\"Golden\", bracket=(8,12))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da804f75-d383-43c7-bfe6-96cfeb73bf6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# But bracket is not a hard constraint. The answer can go out of the braket based on situation\n",
    "optimize.minimize_scalar(objective, method=\"Brent\", bracket=(6,8))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "217b73bd-36a0-4c42-a80b-c7d5a3a2c9e2",
   "metadata": {},
   "source": [
    "<img src=\"https://img.icons8.com/external-itim2101-lineal-color-itim2101/344/external-professor-life-style-avatar-itim2101-lineal-color-itim2101.png\" alt=\"Instructor\" width=50> <font size=8>Multi Variable Optimization</font><br>\n",
    "The `optimize.minimize` function implements a variety of optimization methods. These methods are distinguished by `method` argument. We will review them one by one here. This function works with one or more variable functions. It also covers constrained optimization, which we will review later."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54e32078-25e8-4f49-b65c-1dbe45b70999",
   "metadata": {},
   "source": [
    "<h2> Simplex Search</h2>\n",
    "The first method is a heurstic  and gradient free method called Nelder-Mead method. It is also know as Simplex Search. A simplex is a n+1 points shape in n dimension. For example in 2D it is a triangle. In the begining, the simplex is chosen randomally, and it evolves by iteration. Based on value of the function at a simplex points, we always try to replace the worst one with a new point. The next point is selected from remaining point based on a set of rules. These rules are : Reflection, expansion, contraction and collapse and depicted in the following picture. Based on the value of the function at simplex points one rule is selected. At each step the convergence is checked. One stop condition is checking the standard deviation of value of the function and see if it saturates or not. \n",
    "<center><table><tr><td><img src=\"images/simplex_operation.jpeg\" width=300></td><td><img src=\"https://upload.wikimedia.org/wikipedia/commons/d/de/Nelder-Mead_Himmelblau.gif\" alt=\"demo\" width=400></td><td></td></tr></table></center>\n",
    "Check the following page for more details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53ffcbd6-3575-4e45-b602-a380fdbb40a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Himmelblau's function\n",
    "objective = lambda x: (x[0]**2 +x[1] -11)**2 + (x[0] +x[1]**2 -7)**2\n",
    "x0 = x0=np.array([1,1],dtype=np.float32)\n",
    "optimize.minimize(objective, method='Nelder-Mead', x0=x0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b9e48a1-0c7e-469a-9c9e-7788b5a118a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimize.minimize(objective, method='Nelder-Mead', x0=x0, options={'maxiter':1000, 'xatol' : 1.E-6})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5848a926-9eda-4bfa-af84-17e3a2f2c37e",
   "metadata": {},
   "source": [
    "<h2> Powell Method</h2>\n",
    "Exteded version of Brent method for multi-variable function is called Powell method.\n",
    "<a href=\"https://en.wikipedia.org/wiki/Powell%27s_method\">Check here for more details</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "528f6970-0ae3-49f7-bd94-2d40294a270b",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimize.minimize(objective, method='Powell', x0=x0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac6bfd4d-b351-4465-add9-4614b24b5585",
   "metadata": {
    "tags": []
   },
   "source": [
    "<h2> Conjugate Gradient</h2>\n",
    "This method as the name suggests, it is based on gradient but it is not the same as gradient descent. CG is a method for solving the linear equation systems(<a href=\"https://en.wikipedia.org/wiki/Conjugate_gradient_method\">See here for details</a>). You can consider CG as finding the solution of equation ▽f(x) = 0. The method has much details and it convert the optimization problem to an orthogonal decomposition of gradient. Check <a href=\"https://www.youtube.com/watch?v=h4cG8jLGmKg\"> this video</a> for learning about the math underhood of this method. The method can be applied to unconstrained convex optimizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4150412e-c9b1-4c32-8c7e-b8fd6ee1477e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "optimize.minimize(objective, method=\"CG\", x0=x0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4840517-ec09-47a2-9eba-cfabea16cafd",
   "metadata": {},
   "source": [
    "<img src=\"https://img.icons8.com/color/344/light.png\" width=50> Scipy has an internal implementation of numerical differentiation. Check this function for more details `scipy.optimize.approx_fprime`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c748c45f-455c-4e51-be17-8db3996a4e6b",
   "metadata": {},
   "source": [
    "<img src= \"https://img.icons8.com/external-flaticons-flat-flat-icons/344/external-question-100-most-used-icons-flaticons-flat-flat-icons.png\" alt=\"Tip\" width=60> __Question__:<br>\n",
    "There are a set of functions for comparing the optimization algorithms. Check the following wikipedia page for a list of some of them:<br>\n",
    "<a href=\"https://en.wikipedia.org/wiki/Test_functions_for_optimization\"> Test functions for optimization</a> <br>\n",
    "Do the followings: <br>\n",
    "<ul>\n",
    "    <li>from the above wiki page find the definition for <em>Beale function</em></li>\n",
    "    <li> write a function for it</li>\n",
    "    <li> find its minimum using the methods we discussed so far. Use a random initialization.</li>\n",
    "    <li> Compare the accuracy of methods</li>\n",
    " </ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b352690c-ed61-40d1-83ae-d4093879e386",
   "metadata": {},
   "source": [
    "<h1> Second Order Methods:</h1>\n",
    "    The second order methods uses hessian matrix in addition to gradient vector. This category contains Newton and Quasi-Newton methods. They can be used when the number optimization parameters are small. (For example for a DNN model with 50 K parameters, the hessian will be 50K x 50K  and have 2500K elements !!). The Newton method specifies the step size in a very efficient way so can be fast but at the same time it requires inversing the hessian matrix. The Quasi-newton method try to optimize the calculations and estimate hessian inverse. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4106c819-d601-44c6-bac5-1a600a5e5a81",
   "metadata": {
    "tags": []
   },
   "source": [
    "<h2>Newton Method</h2>\n",
    "Newton method is based on a second order taylor approximation of the function and solving the optimal point for it. See the following figure for clarification.\n",
    "\n",
    "<tabble><tr><td><img src=\"images/newton_method.jpeg\" width=300></td><td> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  </td><td><img  src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/90e32b708ca17d5659fdc482fe3c9f88996361ba\"></td></tr></table><br>\n",
    "For multi-dimensional optimization the equations is: <br>\n",
    "<img src=\"images/newton_multid.png\" width=500> <br>\n",
    "The function ```optimize.newton```  implements the newton method. \n",
    "<br>But, a better way is using a hessian free implementation called *Truncated Newton*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f602376-5df7-4cc7-bcf1-dddbdd0d0c6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Truncated Newton -> Hessian Free\n",
    "optimize.minimize(objective, method=\"TNC\", x0=x0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f47a0c9-7827-4472-bd33-fdc307fcd00c",
   "metadata": {},
   "source": [
    "<img src=\"https://img.icons8.com/color/344/light.png\" width=50> There is a combinantion of Conjugate Gradinet and Newton algorithms called Newton-CG.<br> You have to provide the jacobian for this method.<br>\n",
    "For using this method run:  ```optimize.minimize(objective, method=\"Newton-CG\", jac = jac, x0=x0)```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82214828-283f-45f0-a126-47ae3375aa85",
   "metadata": {},
   "source": [
    "<h2>Levenberg–Marquardt</h2>\n",
    "The LM algorithm is one of widely used algorithm in optimization (specifically in the context of traditional neural networks). It uses a regularized version of hessian for step size. The update rule is:<br>\n",
    "<img src=\"images/lm.png\" width =400><br>\n",
    "For λ → 0, LM converts to Newton and for λ → ∞ it converts to GD. For using LM, one can call the function:<br>\n",
    "<code>optimize.least_squares</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fd80b6c-bcdd-4aaf-97ab-0890cae4dffb",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimize.least_squares(objective, x0=x0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a282f39c-b6b3-4df8-999f-45efc20ac483",
   "metadata": {},
   "source": [
    "<h1>BFGS Algorithm</h1>\n",
    "Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm is one the most important ones implemented by Scipy. BFGS is used for unconstrained nonlinear optimization. It is faster than Newton method (O(n^2) compared to O(n^3) in newton method). Check <a href=\"https://archive.org/details/practicalmethods0000flet\">this</a> for the mathematics of this algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "929f709d-5453-4cc5-a926-f58939e13f31",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimize.minimize(objective, method=\"BFGS\", x0=x0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d19161f-390a-488a-8bb4-81ae0bf10e10",
   "metadata": {},
   "source": [
    "<h2>L-BFGS-B Algorithm</h2>\n",
    "L-BFGS-B is a limited memory version of BFGS and it is the default method for the minimize function. check this <a href=\"https://en.wikipedia.org/wiki/Limited-memory_BFGS\"> wiki page</a> for the details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c8cdacc-ab5e-4045-a8c1-17d7f37ac1c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimize.minimize(objective, method=\"L-BFGS-B\", x0=x0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64fa9b35-1bda-4b01-9d15-b5d963df9fa4",
   "metadata": {},
   "source": [
    "<img src=\"https://img.icons8.com/external-flaticons-lineal-color-flat-icons/344/external-coffee-cup-bakery-flaticons-lineal-color-flat-icons.png\" alt=\"Takehome\" width=60> <font size=6>Takehome:</font><br>\n",
    "There are few more methods and functions that we have not tried them here. You can do research on your own for learning about them.<br>\n",
    "<ul>\n",
    "    <li> <code>optimize.minimize(objective, method=\"COBYLA\", x0=x0)</code></li>\n",
    "    <li> <code>optimize.minimize(objective, method=\"SLSQP\", x0=x0)</code></li>\n",
    "    <li><code>optimize.dual_annealing</code></li>\n",
    "    <li><code>optimize.basinhopping</code></li>\n",
    "    </ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dac38a4-7151-42e4-8630-3a1173891a78",
   "metadata": {},
   "source": [
    "<img src=\"https://img.icons8.com/external-kosonicon-lineal-color-kosonicon/344/external-lab-tool-back-to-school-kosonicon-lineal-color-kosonicon.png\" alt=\"Lab\" width=80 > <font size=6>Lab (Elastic Demand):</font><br>\n",
    "In elastic demand scenario, price of a product is function of supply. More supply lowers the price and vice versa. See the following figure.\n",
    "<br><img src=\"images/ElasticDemand.png\" width=400><br>\n",
    "Let's suppose that a product has the following demand equation:<br>\n",
    "<img src=\"images/demand.png\" width=250><br>\n",
    "Also, cost of building a unit of product is function of quantity. Let's say for our product, the cost has the following equation:<br>\n",
    "<img src=\"images/cost.png\" width=300><br>\n",
    "Write a program to answer the following questions:\n",
    "    <ul>\n",
    "        <li> At what price the revenue is maximized?</li>\n",
    "        <li> At what price the profit is maximized?</li>\n",
    "    <li> At what price the cost is minimized?</li>\n",
    "    </ul>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ef596aa-0734-4838-9055-84d4c513444f",
   "metadata": {},
   "source": [
    "<h1> Constrained Optimization</h1>\n",
    "What we studied so far was unconstrained optimization, which the solution can be any point in the domain of objective function. But in real life we might have some constraints. These constraints limit the choice of optimization algorithm. We can equality and non-equality constraints. The genral optimization problem has the following form:<br>\n",
    "<img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/7b8beab031562d937314a4894ec449189f179219\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "967c1884-9806-4eea-9014-f258d579f2ed",
   "metadata": {
    "tags": []
   },
   "source": [
    "<table><tr><td><img src=\"https://upload.wikimedia.org/wikipedia/commons/4/46/%D0%9B%D0%B0%D0%B3%D1%80%D0%B0%D0%BD%D0%B6.jpg\" width=200></td><td><h2> Lagrange Multipliers for Constrained Opimization</h2>\n",
    "<br> There is a mathematical trick to convert every  (equality) constrained optimization problem to a non-constrained one. This trick is attributed to Lagrange. In this trick we replace the objective function with a new version of it called Lagrangian and we add some new parameters to the function they are called lagrangian multipliers. <br>\n",
    "Check this document for more details: <a href=\"https://en.wikipedia.org/wiki/Lagrange_multiplier\">Lagrange_multiplier</a></td></tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43982472-d180-47bb-bed4-958e8c51c972",
   "metadata": {},
   "source": [
    "Many of above discussed methods support constrained optimization. Let's review scipy's capablities for constrained optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5342dfc-943c-4e83-9f05-85440b1fd686",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining bounds for the solution\n",
    "x0 = np.array([0,0], dtype=np.float32)\n",
    "bounds = [(-4, 1), (-4, 1)] \n",
    "optimize.minimize(objective,  x0=x0, bounds=bounds,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50c36eba-b39d-4d17-b57e-cd00a68cfee2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check how sensitive it is to initial condition\n",
    "x0 = np.array([-1,-1], dtype=np.float32) \n",
    "bounds = [(-4, 1), (-4, 1)] \n",
    "optimize.minimize(objective, x0=x0, bounds=bounds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7b2fea2-0f2d-4af1-a92a-b558c12e96a8",
   "metadata": {},
   "source": [
    "<h4> Defining linear constraints</h4>\n",
    "We can have linear equality constraints like: <br>\n",
    "<img src=\"images/linear_constraints.png\" width = 400>\n",
    "This can be written as Ax=0 where:<br>\n",
    "<img src=\"images/linear_constraints_mat.png\" width=300>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37392102-68e5-4aa6-bf2d-30156737c1b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "x0 = np.array([-1,-1], dtype=np.float32) \n",
    "lin_cond = np.array([[1, 1],[2, 1]], dtype=np.float32)\n",
    "lower_bound = (-4, -4)\n",
    "upper_bound = (1,1)\n",
    "lin_const = optimize.LinearConstraint(lin_cond, lower_bound, upper_bound)\n",
    "optimize.minimize(objective, x0=x0, constraints=lin_const)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3e8383f-59d1-421a-8234-8addb394e109",
   "metadata": {},
   "source": [
    "<h4>Arbitrary Constraint</h4>\n",
    "You can define a function as a constraint. Also, non equality constraint can be covered this way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1303271-9585-454b-9cd3-acb99beba6df",
   "metadata": {},
   "outputs": [],
   "source": [
    "cons = {\"type\":\"eq\", \"fun\":lambda x: 4*x[0] + 3*x[1]}\n",
    "x0 = np.array([-1,-1], dtype=np.float32) \n",
    "\n",
    "lin_const = optimize.LinearConstraint(lin_cond, lower_bound, upper_bound)\n",
    "optimize.minimize(objective, x0=x0, constraints=cons)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be25beab-4183-46b8-aa32-84869a4c2d6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "cons = [{\"type\":\"eq\", \"fun\":lambda x: 4*x[0] + 3*x[1]}, {\"type\":\"ineq\", \"fun\":lambda x:x[0]*x[1]}]\n",
    "x0 = np.array([-1,-1], dtype=np.float32) \n",
    "\n",
    "lin_const = optimize.LinearConstraint(lin_cond, lower_bound, upper_bound)\n",
    "optimize.minimize(objective, x0=x0, constraints=cons)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}