Merge branch 'unifyai:master' into master

.github/workflows/manual-tests.yml

@@ -38,7 +38,7 @@ jobs:
            pip install pymongo
            cd ivy
            python setup_tests.py ${{ github.event.inputs.test }} 
-           python run_tests.py ${{ secrets.REDIS_CONNECTION_URL }} ${{ secrets.REDIS_PASSWORD }} ${{ secrets.MONGODB_PASSWORD }} ${{ github.event.inputs.version}} ${{ steps.jobs.outputs.html_url }}
+           python run_tests.py ${{ secrets.REDIS_CONNECTION_URL }} ${{ secrets.REDIS_PASSWORD }} ${{ secrets.MONGODB_PASSWORD }} ${{ github.event.inputs.version}} ${{ github.run_id }} ${{ steps.jobs.outputs.html_url }}
          continue-on-error: true
 
        - name: Check on failures

docker/rebuild_all_dockerfiles.sh

@@ -1,5 +1,4 @@
 #!/bin/bash
 
 docker build -t unifyai/ivy:latest --no-cache -f Dockerfile ..
-docker build -t unifyai/ivy:latest-gpu --no-cache DockerfileGPU ..
-docker build -t unifyai/ivy:latest-copsim --no-cache DockerfileCopsim ..
+docker build -t unifyai/ivy:latest-gpu --no-cache -f DockerfileGPU ..

docs/partial_source/deep_dive/arrays.rst

@@ -24,7 +24,8 @@ Arrays
 .. _`arrays channel`: https://discord.com/channels/799879767196958751/933380487353872454
 .. _`arrays forum`: https://discord.com/channels/799879767196958751/1028296936203235359
 .. _`wrapped logic`: https://github.com/unifyai/ivy/blob/6a729004c5e0db966412b00aa2fce174482da7dd/ivy/func_wrapper.py#L95
-
+.. _`NumPy's`: https://numpy.org/doc/stable/user/basics.dispatch.html#basics-dispatch
+.. _`PyTorch's`: https://pytorch.org/docs/stable/notes/extending.html#extending-torch
 There are two types of array in Ivy, there is the :class:`ivy.NativeArray` and also the :class:`ivy.Array`.
 
 Native Array
@@ -105,6 +106,68 @@ For the reasons explained above, this would be a problem.
 
 Therefore, all compositional functions have a separate piece of `wrapped logic`_ to ensure that all :class:`ivy.NativeArray` instances are converted to :class:`ivy.Array` instances before entering into the compositional function.
 
+Integrating custom classes with Ivy
+-----------------------------------
+
+Ivy's functional API and its functions can easily be integrated with non-Ivy classes. Whether these classes are ones that inherit from Ivy or completely standalone custom classes, using Ivy's :code:`__array_function__`, Ivy's functions can handle inputs of those types.
+
+To make use of that feature, the class must contain an implementation for these functions and it must contain an implementation for the function :code:`__array_function__`. If a non-Ivy class is passed to an Ivy function, a call to this class's :code:`__array_function__` is made which directs Ivy's function to handle that input type correctly. This allows users to define custome implementations for any of the functions that can be found in Ivy's functional API which would further make it easy to integrate those classes with other Ivy projects.
+
+**Note**
+This functionality is inspired by `NumPy's`_ :code:`__array_function__` and `PyTorch's`_ :code:`__torch_function__`. 
+
+As an example, consider the following class :code:`MyArray` with the following definition:
+
+.. code-block:: python
+>>> class MyArray:
+>>>	def __init__(self, data=None):
+>>>		self.data = data
+
+Running any of Ivy’s functions using a :code:`MyArray` object as input will throw an :code:`IvyBackendException` since Ivy’s functions do not support this class type as input. This is where :code:`__array_function__` comes into play. Let’s add the method to our :code:`MyArray` class to see how it works.
+
+There are different ways to do so. One way is to use a global dict :code:`HANDLED_FUNCTIONS`  which will map Ivy’s functions to the custom variant functions:
+
+.. code-block:: python
+>>> HANDLED_FUNCTIONS = {}
+>>> class MyArray:
+>>>	def __init__(self, data=None):
+>>>		self.data = data
+>>>	def __array_function__(self, func, types, args, kwargs):
+>>>		if func not in HANDLED_FUNCTIONS:
+>>>			return NotImplemented
+>>>		if not all((t, (MyArray, ivy.Array, ivy.NativeArray)) for t in types):
+>>>			return NotImplemented
+>>>		return HANDLED_FUNCTIONS[func](*args, **kwargs)		
+
+:code:`__array_function__` accepts four parameters: :code:`func` representing a reference to the array API function being 
+overridden, :code:`types` a list of the types of objects implementing :code:`__array_function__`, :code:`args` a tuple of arguments supplied to the function, and :code:`kwargs` being a dictionary of keyword arguments passed to the function.
+While this class contains an implementation for :code:`__array_function__`, it is still not enough as it is necessary to implement any needed Ivy functions with the new :code:`MyArray` class as input(s) for the code to run successfully.
+We will define a decorator function :code:`implements` that can be used to add functions to :code:`HANDLED_FUNCTIONS`: 
+
+.. code-block:: python
+>>> def implements(ivy_function):
+>>> 	def decorator(func):
+>>>		HANDLED_FUNCTIONS[ivy_function] = func
+>>>		return func
+>>>	return decorator		
+
+Lastly, we need to apply that decorator to the override function. Let’s consider for example a function that overrides :code:`ivy.abs`:
+
+.. code-block:: python
+>>> @implements(ivy.abs)
+>>> def my_abs(my_array, ivy_array):
+>>> 	my_array.data = abs(my_array.data)
+
+Now that we have added the function to :code:`HANDLED_FUNCTIONS`, we can now use :code:`ivy.abs` with :code:`MyArray` objects:
+.. code-block:: python
+
+>>>	X = MyArray(-3)
+>>>	X = ivy.abs(X)
+
+Of course :code:`ivy.abs` is an example of a function that is easy to override since it only requires one operand. The same approach can be used to override functions with multiple operands, including arrays or array-like objects that define :code:`__array_function__`. 
+
+It is relevant to mention again that any function not stored inside the dict :code:`HANDLED_FUNCTIONS` will not work and it is also important to notice that the operands passed to the function must match that of the function stored in the dict. For instance :code:`my_abs` takes only one parameter which is a :code:`MyArray` object. So, passing any other operands to the function will result in an exception :code:`IvyBackendException` being thrown. Lastly, for a custom class to be covered completely with Ivy's functional API, it is necessary to create an implementation for all the relevant functions within the API that will be used by this custom class. That can be all the functions in the API or only a subset of them.
+
 **Round Up**
 
 This should have hopefully given you a good feel for the different types of arrays, and how these are handled in Ivy.
@@ -118,4 +181,4 @@ If you have any questions, please feel free to reach out on `discord`_ in the `a
 
     <iframe width="420" height="315"
     src="https://www.youtube.com/embed/tAlDPnWcLDE" class="video">
-    </iframe>
+    </iframe>

ivy/__init__.py

@@ -24,9 +24,13 @@
     import jaxlib
 except ImportError:
     jax = SimpleNamespace()
+    jax.interpreters = SimpleNamespace()
+    jax.interpreters.xla = SimpleNamespace()
     jax.interpreters.xla._DeviceArray = SimpleNamespace()
+    jaxlib = SimpleNamespace()
+    jaxlib.xla_extension = SimpleNamespace()
     jaxlib.xla_extension.DeviceArray = SimpleNamespace()
-    jax.Buffer = SimpleNamespace()
+    jaxlib.xla_extension.Buffer = SimpleNamespace()
 
 warnings.filterwarnings("ignore", module="^(?!.*ivy).*$")
 
@@ -203,7 +207,7 @@ def __new__(cls, shape_tup):
                 torch.Size,
                 jax.interpreters.xla._DeviceArray,
                 jaxlib.xla_extension.DeviceArray,
-                jax.Buffer,
+                jax.xla_extension.Buffer,
                 np.ndarray,
                 tf.Tensor,
             )
@@ -302,6 +306,7 @@ class Node(str):
 array_decimal_values_stack = list()
 warning_level_stack = list()
 nan_policy_stack = list()
+dynamic_backend_stack = list()
 warn_to_regex = {"all": "!.*", "ivy_only": "^(?!.*ivy).*$", "none": ".*"}
 
 
@@ -838,7 +843,6 @@ class GlobalsDict(dict):
         "valid_dtypes": valid_dtypes,
         "valid_numeric_dtypes": valid_numeric_dtypes,
         "valid_int_dtypes": valid_int_dtypes,
-        "valid_int_dtypes": valid_int_dtypes,
         "valid_uint_dtypes": valid_uint_dtypes,
         "valid_complex_dtypes": valid_complex_dtypes,
         "valid_devices": valid_devices,
@@ -862,6 +866,7 @@ class GlobalsDict(dict):
         "default_int_dtype_stack": data_type.default_int_dtype_stack,
         "default_uint_dtype_stack": data_type.default_uint_dtype_stack,
         "nan_policy_stack": nan_policy_stack,
+        "dynamic_backend_stack": dynamic_backend_stack,
     }
 )
 
@@ -1094,3 +1099,55 @@ def unset_nan_policy():
     global nan_policy_stack
     if nan_policy_stack:
         nan_policy_stack.pop(-1)
+
+
+# Dynamic Backend
+
+
+def get_dynamic_backend():
+    """Returns the current dynamic backend setting, with the default being True"""
+    global dynamic_backend_stack
+    if not dynamic_backend_stack:
+        return True
+    else:
+        return dynamic_backend_stack[-1]
+
+
+def set_dynamic_backend(flag):
+    """Sets the global dynamic backend setting to the provided flag (True or False)"""
+    global dynamic_backend_stack
+    if flag not in [True, False]:
+        raise ValueError("dynamic_backend must be a boolean value (True or False)")
+    dynamic_backend_stack.append(flag)
+
+
+def unset_dynamic_backend():
+    """
+    Removes the current dynamic backend setting,
+    restoring the previous setting (if any)
+    """
+    global dynamic_backend_stack
+    if dynamic_backend_stack:
+        dynamic_backend_stack.pop()
+
+
+# Context Managers
+
+
+class DynamicBackendContext:
+    def __init__(self, value):
+        self.value = value
+        self.original = None
+
+    def __enter__(self):
+        self.original = get_dynamic_backend()
+        set_dynamic_backend(self.value)
+
+    def __exit__(self, type, value, traceback):
+        unset_dynamic_backend()
+        if self.original is not None:
+            set_dynamic_backend(self.original)
+
+
+def dynamic_backend_as(value):
+    return DynamicBackendContext(value)

ivy/array/array.py

@@ -72,7 +72,7 @@ class Array(
     ArrayWithStatisticalExperimental,
     ArrayWithUtilityExperimental,
 ):
-    def __init__(self, data):
+    def __init__(self, data, dynamic_backend=None):
         ArrayWithActivations.__init__(self)
         ArrayWithCreation.__init__(self)
         ArrayWithDataTypes.__init__(self)
@@ -112,9 +112,9 @@ def __init__(self, data):
         ArrayWithSortingExperimental.__init__(self),
         ArrayWithStatisticalExperimental.__init__(self),
         ArrayWithUtilityExperimental.__init__(self),
-        self._init(data)
+        self._init(data, dynamic_backend)
 
-    def _init(self, data):
+    def _init(self, data, dynamic_backend=None):
         if ivy.is_ivy_array(data):
             self._data = data.data
         else:
@@ -135,10 +135,46 @@ def _init(self, data):
         else:
             self._post_repr = ")"
         self.backend = ivy.current_backend_str()
+        if dynamic_backend is not None:
+            self._dynamic_backend = dynamic_backend
+        else:
+            self._dynamic_backend = ivy.get_dynamic_backend()
 
     # Properties #
     # ---------- #
 
+    @property
+    def dynamic_backend(self):
+        return self._dynamic_backend
+
+    @dynamic_backend.setter
+    def dynamic_backend(self, value):
+        from ivy.functional.ivy.gradients import _variable
+        from ivy.backend_handler import _determine_backend_from_args
+
+        if value == False:
+            self._backend = _determine_backend_from_args(self)
+
+        else:
+            is_variable = self._backend.is_variable
+            to_numpy = self._backend.to_numpy
+            variable_data = self._backend.variable_data
+
+            if is_variable(self.data) and not (
+                str(self._backend).__contains__("jax")
+                or str(self._backend).__contains__("numpy")
+            ):
+                native_data = variable_data(self.data)
+                np_data = to_numpy(native_data)
+                new_arr = ivy.array(np_data)
+                self._data = _variable(new_arr).data
+
+            else:
+                np_data = to_numpy(self.data)
+                self._data = ivy.array(np_data).data
+
+        self._dynamic_backend = value
+
     @property
     def data(self) -> ivy.NativeArray:
         """The native array being wrapped in self."""
@@ -217,6 +253,24 @@ def __torch_function__(cls, func, types, args=(), kwargs={}):
         args, kwargs = args_to_native(*args, **kwargs)
         return func(*args, **kwargs)
 
+    def __array_function__(self, func, types, args, kwargs):
+        # Cannot handle items that have __array_function__ other than those of
+        # ivy arrays or native arrays.
+        for t in types:
+            if (
+                hasattr(t, "__array_function__")
+                and (t.__array_function__ is not ivy.Array.__array_function__)
+                or (
+                    hasattr(ivy.NativeArray, "__array_function__")
+                    and (t.__array_function__ is not ivy.NativeArray.__array_function__)
+                )
+            ):
+                return NotImplemented
+
+        # Arguments contain no overrides, so we can safely call the
+        # overloaded function again.
+        return func(*args, **kwargs)
+
     def __array__(self, *args, **kwargs):
         args, kwargs = args_to_native(*args, **kwargs)
         return self._data.__array__(*args, **kwargs)

ivy/array/experimental/activations.py

@@ -80,3 +80,32 @@ def thresholded_relu(
         ivy.array([0., 0., 1.])
         """
         return ivy.thresholded_relu(self._data, threshold=threshold, out=out)
+
+    def prelu(
+        self,
+        slope: Union[float, ivy.NativeArray, ivy.Array],
+        /,
+        *,
+        out: Optional["ivy.Array"] = None,
+    ) -> ivy.Array:
+        """
+        Prelu takes input data (Array) and slope array as input,
+        and produces one output data (array) where the function
+        f(x) = slope * x for x < 0, f(x) = x for x >= 0., is applied
+        to the data array elementwise. This operator supports unidirectional
+        broadcasting (array slope should be unidirectional broadcastable to
+        input tensor X);
+
+        Parameters
+        ----------
+        self
+            input array.
+        slope
+            Slope Array. The shape of slope can be smaller then first input X;
+            if so, its shape must be unidirectional broadcastable to X.
+        out
+            Optional output array.
+        Returns
+        -------
+        """
+        return ivy.prelu(self._data, slope, out=out)