96617 dev review status

KNOWLG-SETUP.md

@@ -0,0 +1,125 @@
+
+Below are the steps to set up the Sunbird Knowlg Microservices, DBs with seed data and Jobs. It uses a local Kubernetes cluster deploy the required services.
+
+### Prerequisites:
+* Java 11
+* Maven
+* Docker
+* Minikube - It implements a local Kubernetes cluster on macOS, Linux, and Windows.
+* KubeCtl - The Kubernetes command-line tool
+
+### Prepare folders for database data and logs
+
+```shell
+mkdir -p ~/sunbird-dbs/neo4j ~/sunbird-dbs/cassandra ~/sunbird-dbs/redis ~/sunbird-dbs/es ~/sunbird-dbs/kafka
+export sunbird_dbs_path=~/sunbird-dbs
+```
+
+
+
+### Docker Images of Knowlg MicroServices
+Start Docker in your machine and create the Docker Images of below microservices using the shell script.
+1. taxonomy-service
+2. content-service
+3. search-service
+
+```shell
+sh ./knowlg-docker-image.sh <TAG> # provide the TAG for the docker image.
+```
+**Note:** Please specify the TAG for the Docker Images and update the configuration in helm chart of respective deployment.
+
+Check the Docker Images
+```shell
+docker image ls -a
+```
+**Output:**
+```shell
+❯❯ docker image ls -a
+REPOSITORY                    TAG       IMAGE ID       CREATED          SIZE
+assessment-service            R5.0.0    72a9cc1b2cc4   14 seconds ago   479MB
+search-service                R5.0.0    24b7d8947a4f   23 seconds ago   465MB
+content-service               R5.0.0    afcbc9c10fa3   33 seconds ago   556MB
+taxonomy-service              R5.0.0    a8a24a6241f2   47 seconds ago   480MB
+```
+
+### Kubernetes Cluster Setup
+Please use the minikube to quickly set up the kubernetes cluster in local machine.
+
+```shell
+minikube start
+```
+
+### Load Docker Images to Minikube Cluster
+```shell
+minikube image load neo4j:3.3.0
+minikube image load taxonomy-service:R5.0.0
+```
+
+### Create Namespace 
+Create the namespaces to deploy the API microservices, DBs and Jobs.
+1. knowlg-api
+2. knowlg-db
+3. knowlg-job
+
+```shell
+kubectl create namespace knowlg-api
+kubectl create namespace knowlg-db
+kubectl create namespace knowlg-job
+```
+
+### Setup Databases
+Please run the below `helm` commands to set up the required databases within the kubernets cluster.
+It requires the below DBs for Knowlg.
+1. Neo4J
+2. Cassandra
+3. Elasticsearch
+4. Kafka
+5. Redis
+
+```shell
+cd kubernetes
+helm install redis sunbird-dbs/redis -n knowlg-db
+
+minikube mount <LOCAL_SOURCE_DIR>:/var/lib/neo4j/data // LOCAL_SOURCE_DIR is where neo4j dump is extracted Ex: /Users/abc/sunbird-dbs/neo4j/data
+helm install neo4j sunbird-dbs/neo4j -n knowlg-db
+
+minikube mount <LOCAL_SOURCE_DIR>:/mnt/backups // LOCAL_SOURCE_DIR is where neo4j dump is extracted Ex: /Users/abc/sunbird-dbs/cassandra/backups
+helm install cassandra sunbird-dbs/cassandra -n knowlg-db
+
+ssh to cassandra pod
+run => cqlsh
+run => source '/mnt/backups/cassandra_backup/db_schema.cql';
+```
+
+**Note:** 
+- The `helm` charts for Kafka, Elasticsearch will be added soon.
+
+### Define ConfigMap
+We use the configmap to load the configuration for the microservices.
+
+#### ConfigMap for Taxonomy-Service
+Use the below commands to load the configmap of taxonomy-service.
+1. `taxonomy-config` - this has the application configuration. Please update the variables with respect to your context and load.
+2. `taxonomy-xml-config` - this has the logback configuration to handle the logs.
+
+We have to update the below configurations in `taxonomy/templates/taxonomy-service_application.conf` specific to your context.
+
+```shell
+cd kubernetes
+kubectl create configmap taxonomy-xml-config --from-file=taxonomy/taxonomy-service_logback.xml -n knowlg-api -o=yaml
+kubectl create configmap taxonomy-config --from-file=taxonomy/taxonomy-service_application.conf -n knowlg-api  -o=yaml
+```
+
+### Run Taxonomy-Service
+Use the `taxonomy` helm chart to run the taxonomy-service in local kubernetes cluster.
+
+```shell
+cd kubernetes
+helm install taxonomy taxonomy -n knowlg-api
+```
+Use Port Forwarding to access the application in the cluster from local.
+
+```shell
+kubectl port-forward <pod-name> 9000:9000 -n knowlg-api
+curl 'localhost:9000/health'
+```

README.md

@@ -2,13 +2,32 @@
 
 Repository for Knowledge Platform - 2.0
 
-## Knowledge-platform local setup
+## Knowledge-platform local setup 
 This readme file contains the instruction to set up and run the content-service in local machine.
 
 ### System Requirements:
 
 ### Prerequisites:
 * Java 11
+* Docker, Docker Compose
+
+
+## One step installation 
+
+1. Go to Root folder (knowledge-platform)
+2. Run "local-setup.sh" file
+``` shell
+sh ./local-setup.md
+```
+
+ This will install all the requied dcoker images & local folders for DB mounting.
+ 3. Follow the below manual setps of running content service 
+  refer: [Running Content Service:](#running-content-service)
+
+
+
+## Manual steps to install all the dependents
+Please follow the manual steps in [One step installation](#one-step-installation) is failed.
 
 ### Prepare folders for database data and logs
 

content-api/collection-csv-actors/src/main/scala/org.sunbird/collectioncsv/manager/CollectionCSVManager.scala

@@ -29,10 +29,15 @@ import scala.collection.mutable
 import scala.collection.mutable.ListBuffer
 import scala.concurrent.{ExecutionContext, Future}
 
+import scala.collection.JavaConverters._
+
 object CollectionCSVManager extends CollectionInputFileReader  {
 
   private val CONTENT_FOLDER = "cloud_storage.content.folder"
   val logger = Logger.getLogger("CollectionCSVManager")
+  val categoryMap: java.util.Map[String, AnyRef] = Platform.getAnyRef("contentTypeToPrimaryCategory",
+    new util.HashMap[String, AnyRef]()).asInstanceOf[java.util.Map[String, AnyRef]]
+
   def getCode(code: String): String = {DigestUtils.md5Hex(code)}
 
   def validateInputData(inputFileExtension: String, csvRecords: util.List[CSVRecord], mode: String, collectionHierarchy: Map[String, AnyRef])(implicit oec: OntologyEngineContext, ec: ExecutionContext): List[Map[String, AnyRef]] = {
@@ -72,14 +77,14 @@ object CollectionCSVManager extends CollectionInputFileReader  {
     val updateHierarchyResponse = UpdateHierarchyManager.updateHierarchy(getUpdateHierarchyRequest(nodesMetadata, hierarchyMetadata))
     TelemetryManager.info(s"CollectionCSVManager:updateCollection --> identifier: ${collectionHierarchy(CollectionTOCConstants.IDENTIFIER).toString} -> after invoking updateHierarchyManager: " + updateHierarchyResponse)
 
-    // Invoke DIAL code linking if mode=UPDATE
-    if(mode.equals(CollectionTOCConstants.UPDATE)) {
+//    // Invoke DIAL code linking if mode=UPDATE
+    updateHierarchyResponse.map{ res => if(mode.equals(CollectionTOCConstants.UPDATE)) {
       linkDIALCodes(folderInfoMap, collectionHierarchy(CollectionTOCConstants.CHANNEL).toString, collectionHierarchy(CollectionTOCConstants.IDENTIFIER).toString)
+       res
+      } else res
     }
-
-    updateHierarchyResponse
   }
-
+  
   def createCSVFileAndStore(collectionHierarchy: Map[String, AnyRef], collectionTocFileName: String)(implicit ss: StorageService): String = {
     val collectionName = collectionHierarchy(CollectionTOCConstants.NAME).toString
     val collectionType = collectionHierarchy.getOrElse(CollectionTOCConstants.CONTENT_TYPE,"").toString
@@ -282,7 +287,6 @@ object CollectionCSVManager extends CollectionInputFileReader  {
                   .equalsIgnoreCase(CollectionTOCConstants.YES)) CollectionTOCConstants.YES else CollectionTOCConstants.NO
 
                 val dialCode = if(csvRecordMap(CollectionTOCConstants.QR_CODE).nonEmpty) csvRecordMap(CollectionTOCConstants.QR_CODE).trim else ""
-
                 val csvLinkedContentsList: Seq[String] = csvRecord.toMap.asScala.toMap.map(colData => {
                   if(linkedContentHdrColumnsList.contains(colData._1) && colData._2.nonEmpty) colData._2.trim.toLowerCase() else ""
                 }).filter(msg => msg.nonEmpty).toSeq
@@ -323,9 +327,9 @@ object CollectionCSVManager extends CollectionInputFileReader  {
       else
         try {
           s""""${nodeInfo(CollectionTOCConstants.IDENTIFIER).toString}": {"isNew": false,"root": false, "metadata": {"mimeType": "application/vnd.ekstep.content-collection",
-             |"contentType": "$collectionUnitType","name": ${JsonUtils.serialize(nodeInfo("name").toString.trim)},
+             |"contentType": "$collectionUnitType","name": ${JsonUtils.serialize(nodeInfo("name").toString.trim)}, "primaryCategory": "${getPrimaryCategory(collectionUnitType)}",
              |"description": ${if(nodeInfo.contains(CollectionTOCConstants.DESCRIPTION)) JsonUtils.serialize(nodeInfo(CollectionTOCConstants.DESCRIPTION).toString) else JsonUtils.serialize("")},
-             |"dialcodeRequired": "${nodeInfo(CollectionTOCConstants.DIAL_CODE_REQUIRED).toString}","dialcodes": "${nodeInfo(CollectionTOCConstants.DIAL_CODES).toString}",
+             |"dialcodeRequired": "${nodeInfo(CollectionTOCConstants.DIAL_CODE_REQUIRED).toString}","dialcodes": ["${nodeInfo(CollectionTOCConstants.DIAL_CODES).toString}"],
              |"code": "${nodeInfo(CollectionTOCConstants.IDENTIFIER).toString}","framework": "$frameworkID",
              |"keywords": ${if(nodeInfo.contains(CollectionTOCConstants.KEYWORDS) && nodeInfo(CollectionTOCConstants.KEYWORDS).asInstanceOf[List[String]].nonEmpty)
               nodeInfo(CollectionTOCConstants.KEYWORDS).asInstanceOf[List[String]].map(keyword=>JsonUtils.serialize(keyword)).mkString("[",",","]") else "[]"},
@@ -500,5 +504,13 @@ object CollectionCSVManager extends CollectionInputFileReader  {
     })
   }
 
+  private def getPrimaryCategory(contentType: String): String ={
+    val primaryCategory = categoryMap.get(contentType)
+    if(primaryCategory.isInstanceOf[String])
+      primaryCategory.asInstanceOf[String]
+    else
+      primaryCategory.asInstanceOf[util.List[String]].asScala.headOption.getOrElse("Learning Resource")
+
+  }
 
 }

content-api/collection-csv-actors/src/main/scala/org.sunbird/collectioncsv/util/CollectionTOCUtil.scala

@@ -107,6 +107,7 @@ object CollectionTOCUtil {
     }
     val headerParam = new util.HashMap[String, String]{put(CollectionTOCConstants.X_CHANNEL_ID, channelId); put(CollectionTOCConstants.CONTENT_TYPE_HEADER, CollectionTOCConstants.APPLICATION_JSON);}
     val requestUrl = Platform.config.getString(CollectionTOCConstants.LINK_DIAL_CODE_API) + "/" + collectionID
+
     TelemetryManager.log("CollectionTOCUtil --> linkDIALCode --> requestUrl: " + requestUrl)
     TelemetryManager.log("CollectionTOCUtil --> linkDIALCode --> reqMap: " + JsonUtils.serialize(reqMap))
     val linkResponse = oec.httpUtil.post(requestUrl, reqMap, headerParam)

content-api/content-actors/src/main/scala/org/sunbird/content/actors/ContentActor.scala

@@ -26,13 +26,14 @@ import org.sunbird.graph.nodes.DataNode
 import org.sunbird.graph.utils.NodeUtil
 import org.sunbird.managers.HierarchyManager
 import org.sunbird.managers.HierarchyManager.hierarchyPrefix
+import org.slf4j.LoggerFactory
 
 import scala.collection.JavaConverters
 import scala.collection.JavaConverters._
 import scala.concurrent.{ExecutionContext, Future}
 
 class ContentActor @Inject() (implicit oec: OntologyEngineContext, ss: StorageService) extends BaseActor {
-
+  private[this] val logger = LoggerFactory.getLogger(classOf[ContentActor])
 	implicit val ec: ExecutionContext = getContext().dispatcher
 	private lazy val importConfig = getImportConfig()
 	private lazy val importMgr = new ImportManager(importConfig)
@@ -75,15 +76,22 @@ class ContentActor @Inject() (implicit oec: OntologyEngineContext, ss: StorageSe
 		val responseSchemaName: String = request.getContext.getOrDefault(ContentConstants.RESPONSE_SCHEMA_NAME, "").asInstanceOf[String]
 		val fields: util.List[String] = JavaConverters.seqAsJavaListConverter(request.get("fields").asInstanceOf[String].split(",").filter(field => StringUtils.isNotBlank(field) && !StringUtils.equalsIgnoreCase(field, "null"))).asJava
 		request.getRequest.put("fields", fields)
+		logger.info("fields:::::::::::::::::::::::: " + request)
 		DataNode.read(request).map(node => {
+			logger.info("check node::::::::::::: " + node)
 			val metadata: util.Map[String, AnyRef] = NodeUtil.serialize(node, fields, node.getObjectType.toLowerCase.replace("image", ""), request.getContext.get("version").asInstanceOf[String])
+			logger.info("metadata------------------------- " + metadata)
 			metadata.put(ContentConstants.IDENTIFIER, node.getIdentifier.replace(".img", ""))
+			logger.info("metadata img:::::::::::::::::::::::::: " + metadata)
 			val response: Response = ResponseHandler.OK
+			logger.info("response::::::::::::::::::::: " + response)
       if (responseSchemaName.isEmpty) {
         response.put("content", metadata)
+		logger.info("response if::::::::::::::::::: " + response)
       }
       else {
         response.put(responseSchemaName, metadata)
+		logger.info("response else::::::::::::::::::: " + response)
       }
 			if(!StringUtils.equalsIgnoreCase(metadata.get("visibility").asInstanceOf[String],"Private")) {
 				response

content-api/content-actors/src/main/scala/org/sunbird/content/dial/DIALManager.scala

@@ -31,6 +31,7 @@ object DIALManager {
 		val objectId: String = request.getContext.getOrDefault(DIALConstants.IDENTIFIER, "").asInstanceOf[String]
 		val reqList: List[Map[String, List[String]]] = getRequestData(request)
 		val requestMap: Map[String, List[String]] = validateAndGetRequestMap(channelId, reqList)
+
 		linkType match {
 			case DIALConstants.CONTENT => linkContent(requestMap, request.getContext)
 			case DIALConstants.COLLECTION => linkCollection(objectId, requestMap, request.getContext)

content-api/content-actors/src/main/scala/org/sunbird/content/publish/mgr/PublishManager.scala

@@ -78,7 +78,7 @@ object PublishManager {
 		}
 
 		//objectData
-		objectData.put(ContentConstants.ID, identifier)
+		objectData.put(ContentConstants.ID, identifier.replace(".img",""))
 		objectData.put(ContentConstants.VER, metadata.get(ContentConstants.VERSION_KEY))
 
 		//edata
@@ -91,7 +91,7 @@ object PublishManager {
 		instructionEventMetadata.put(ContentConstants.PACKAGE_VERSION, metadata.getOrDefault(ContentConstants.PACKAGE_VERSION,0.asInstanceOf[AnyRef]))
 		instructionEventMetadata.put(ContentConstants.MIME_TYPE, metadata.get(ContentConstants.MIME_TYPE))
 		instructionEventMetadata.put(ContentConstants.LAST_PUBLISHED_BY, metadata.get(ContentConstants.LAST_PUBLISHED_BY))
-		instructionEventMetadata.put(ContentConstants.IDENTIFIER, identifier)
+		instructionEventMetadata.put(ContentConstants.IDENTIFIER, identifier.replace(".img",""))
 		instructionEventMetadata.put(ContentConstants.OBJECT_TYPE, objectType)
 		edata.put(ContentConstants.METADATA, instructionEventMetadata)
 		edata.put(ContentConstants.ACTION, ContentConstants.PUBLISH)

content-api/content-service/conf/application.conf

@@ -748,7 +748,19 @@ collection {
 }
 
 #Request timeout
-actor.timeoutMillisec = 30000 
+actor.timeoutMillisec = 30000
 
 #Index file validation
 isIndexHtmlValidationRequired=true
+
+cloud_storage_type=""
+cloud_storage_key=""
+cloud_storage_secret=""
+cloud_storage_container=""
+cloud_storage_endpoint=""
+
+cloudstorage.metadata.replace_absolute_path=false
+cloudstorage.relative_path_prefix= "CONTENT_STORAGE_BASE_PATH"
+cloudstorage.read_base_path="https://sunbirddev.blob.core.windows.net"
+cloudstorage.write_base_path=["https://sunbirddev.blob.core.windows.net"]
+cloudstorage.metadata.list=["appIcon", "artifactUrl", "posterImage", "previewUrl", "thumbnail", "assetsMap", "certTemplate", "itemSetPreviewUrl", "grayScaleAppIcon", "sourceURL", "variants", "downloadUrl", "streamingUrl", "toc_url", "data", "question", "solutions", "editorState", "media", "pdfUrl"]

content-api/hierarchy-manager/src/main/scala/org/sunbird/managers/UpdateHierarchyManager.scala

@@ -27,8 +27,11 @@ object UpdateHierarchyManager {
     def updateHierarchy(request: Request)(implicit oec: OntologyEngineContext, ec: ExecutionContext): Future[Response] = {
         validateRequest(request)
         val nodesModified: java.util.HashMap[String, AnyRef] = request.getRequest.get(HierarchyConstants.NODES_MODIFIED).asInstanceOf[java.util.HashMap[String, AnyRef]]
+        TelemetryManager.info("UpdateHierarchyManager:: updateHierarchy:: nodesModified: " + nodesModified)
         val hierarchy: java.util.HashMap[String, AnyRef] = request.getRequest.get(HierarchyConstants.HIERARCHY).asInstanceOf[java.util.HashMap[String, AnyRef]]
+        TelemetryManager.info("UpdateHierarchyManager:: updateHierarchy:: hierarchy: " + hierarchy)
         val rootId: String = getRootId(nodesModified, hierarchy)
+        TelemetryManager.info("UpdateHierarchyManager:: updateHierarchy:: rootId: " + rootId)
         request.getContext.put(HierarchyConstants.ROOT_ID, rootId)
         getValidatedRootNode(rootId, request).map(node => {
             getExistingHierarchy(request, node).map(existingHierarchy => {

content-api/hierarchy-manager/src/test/scala/org/sunbird/managers/TestHierarchy.scala

@@ -53,7 +53,7 @@ class TestHierarchy extends BaseSpec {
         request.put("rootId", "do_11283193441064550414")
         request.put("unitId", "do_11283193463014195215")
         request.put("children", util.Arrays.asList("do_11340096165525094411"))
-        request.put("relationalMetadata",mapAsJavaMap(Map("do_11340096165525094411" -> Map("relName" -> "Test Name RM", "keywords" -> Array("Overwriting content Keywords") ))))
+        request.put("relationalMetadata",mapAsJavaMap(Map("do_11340096165525094411" -> Map("name" -> "Test Name RM", "keywords" -> Array("Overwriting content Keywords") ))))
         request.put("mode","edit")
         val future = HierarchyManager.addLeafNodesToHierarchy(request)
         future.map(response => {

content-api/hierarchy-manager/src/test/scala/org/sunbird/managers/TestUpdateHierarchy.scala

@@ -394,7 +394,7 @@ class TestUpdateHierarchy extends BaseSpec {
             "            \t\"b9a50833-eff6-4ef5-a2a4-2413f2d51f6c\": {\n"+
             "            \t\t\"root\": false,\n"+
             "            \t\t\"children\": [\"do_31250856200414822416938\",\"do_11340096165525094411\"],\n"+
-            "            \t\t\"relationalMetadata\": {\n\"do_11340096165525094411\": {\n\"relName\": \"abc\"\n,\"keywords\": [\"test\"]}\n}" +
+            "            \t\t\"relationalMetadata\": {\n\"do_11340096165525094411\": {\n\"name\": \"abc\"\n,\"keywords\": [\"test\"]}\n}" +
             "            \t}\n"+
             "            }"
         JsonUtils.deserialize(hierarchyString, classOf[util.HashMap[String, AnyRef]])