Allow spark driver find shuffle pods in specified namespace (apache-s…

…park-on-k8s#357)

The conf property spark.kubernetes.shuffle.namespace is used to
specify the namesapce of shuffle pods.

In normal cases, only one "shuffle daemonset" is deployed and
shared by all spark pods.

The spark driver should be able to list and watch shuffle pods
in the namespace specified by user.

Note: by default, spark driver pod doesn't have authority to
list and watch shuffle pods in another namespace. Some action
is needed to grant it the authority. For example, below ABAC
policy works.

```
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind":
"Policy", "spec": {"group": "system:serviceaccounts", "namespace":
"SHUFFLE_NAMESPACE",
"resource": "pods", "readonly": true}}
```

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/kubernetes/ShufflePodCache.scala

@@ -37,7 +37,8 @@ private[spark] class ShufflePodCache (
 
   def start(): Unit = {
     // seed the initial cache.
-    val pods = client.pods().withLabels(dsLabels.asJava).list()
+    val pods = client.pods()
+      .inNamespace(dsNamespace).withLabels(dsLabels.asJava).list()
     pods.getItems.asScala.foreach {
       pod =>
         if (Readiness.isReady(pod)) {
@@ -50,6 +51,7 @@ private[spark] class ShufflePodCache (
 
     watcher = client
       .pods()
+      .inNamespace(dsNamespace)
       .withLabels(dsLabels.asJava)
       .watch(new Watcher[Pod] {
         override def eventReceived(action: Watcher.Action, p: Pod): Unit = {