Skip to content

Latest commit

 

History

History
61 lines (33 loc) · 2.21 KB

development-in-intellij.md

File metadata and controls

61 lines (33 loc) · 2.21 KB
layout title permalink
doc
Development in Intellij
/docs/development-in-intellij.html

Apache Eagle can be developed in popular IDE, e.g. Intellij and Eclipse. Here we focus on development in Intellij.

1. Prepare Hadoop environment

Normally HDP sandbox is needed for testing Hadoop monitoring. Please reference Quick Start for setting up HDP sandbox.

Please ensure following daemons are up

  • Namenode

  • Datanode

  • HBase master/region server only needed when HBase is metadata store

  • MySQL only neede when MySQL is metadata store

  • Zookeeper

  • Kafka

2. Start Eagle web service in Intellij

Import source code into Intellij, and find eagle-webservice project. Intellij Ultimate supports launching J2EE server within Intellij. If you don't have Intellij Ultimate version, Eclipse is another option.

  • Check service configuration

Under eagle-webservice/src/main/resources, configure application.conf for metadata store. As of Eagle 0.4.0, by default, it points to sandbox HBase. If you want to use MySQL as metadata store, just copy application-mysql.conf to application.conf.

  • Ensure tables are created

Run eagle-service-init.sh under eagle-assembly/src/main/bin/eagle-service-init.sh to create metadata tables.

  • Start Eagle service

Configure Intellij for running tomcat server with eagle-service artifacts

3. Start topology in Intellij

  • Check topology configuration

Take HDFS data activity monitoring as example, config application.conf under eagle-security/eagle-security-hdfs-securitylog/src/main/resources for eagle service endpoint, Kafka endpoint etc.

  • Start Storm topology

Normally one data source has one correponding Storm topology, for example to monitor HDFS data activity, org.apache.eagle.security.auditlog.HdfsAuditLogProcessorMain is used for starting Storm topology and reading audit logs from Kafka.

Other example data integrations are as follows

Hive query activity monitoring : org.apache.eagle.security.hive.jobrunning.HiveJobRunningMonitoringMain

HBase data activity monitoring: org.apache.eagle.security.hbaseHbaseAuditLogMonitoringTopology

MapR FS data activity monitoring: org.apache.eagle.security.auditlog.MapRFSAuditLogProcessorMain