CHANGES.txt

Apache Falcon (incubating) Change log

Trunk (Unreleased)

  INCOMPATIBLE CHANGES

  NEW FEATURES
   FALCON-263 API to get workflow parameters. (pavan kumar kolamuri via Shwetha GS)

  IMPROVEMENTS
   FALCON-554 Extend time functions available in Falcon to support current & 
   last week (Contributed by Srikanth Sundarrajan)

   FALCON-369 Refactor workflow builder. (Shwetha GS)

   FALCON-280 Validate the ACL in Feed entity with the user submitting the entity
   (Jean-Baptiste Onofré via Venkatesh Seetharam)

   FALCON-445 Propagate hive credentials defined in cluster entity to
   hive-site.xml (Sowmya Ramesh via Venkatesh Seetharam)

   FALCON-133 Upgrade to slf4j 1.7.5 and use SLF4J logger. (Jean-Baptiste Onofré
   via Shwetha GS)

   FALCON-353 enable dry run feature of oozie for schedule 
   and update (Shwetha GS via Suhas Vasu)

  OPTIMIZATIONS

  BUG FIXES
   FALCON-556 Feed Replication _SUCCESS is not getting created on target directory.
   (pavan kumar kolamuri via Suhas Vasu)

   FALCON-497 Able to submit feed even though owner of storage specified (location type=data) 
   is different from the ACL owner. (Shwetha GS)

   FALCON-357 HCatalog Feed replication: Hive export job fails when table partition 
   contains multiple dated columns. (Satish Mittal via Shwetha GS)

   FALCON-495 multi source single target feed replication failing in regression.
   (Satish Mittal via Shwetha GS)

   FALCON-494 update feature broken. (Shwetha GS via Suhas Vasu)

   FALCON-496 Feed Replication with Empty Directories giving error. 
   (pavan kumar kolamuri via Shwetha GS)

   FALCON-483 Fix the failing test ConfigurationStoreTest.testConcurrentRemoves 
   on jenkins. (Sowmya Ramesh via Shwetha GS)

   FALCON-430 Process update with user (Shwetha GS via Venkatesh Seetharam)

   FALCON-460 Concurrent deletion of same entity results in NPE (Sowmya Ramesh
   via Venkatesh Seetharam)

   FALCON-459 Lineage resource API fails with NPE for bad query params
   (Sowmya Ramesh via Venkatesh Seetharam)

   FALCON-456 Custom DistCp conflict with core DistCp in container classpath
   fails falcon workflows (Venkatesh Seetharam)

   FALCON-446 Hive Replications fail because of permissions issue
   (Venkatesh Seetharam)

   FALCON-444 Logs dir for replication workflow is incorrect and jobs fail
   with permission issues (Venkatesh Seetharam)

   FALCON-443 Process with Hive workflow engine and filesystem input feeds,
   table output feed fails (Sowmya Ramesh via Venkatesh Seetharam)

   FALCON-382 Error While building Latest trunk code with Hadoop 2.2.0. (Suhas Vasu)

   FALCON-240 Instance status from CLI on a feed doesn't give the retention details. 
   (pavan kumar kolamuri via Shwetha GS)

   FALCON-441 Lineage capture fails for feeds with multiple instances
   (Venkatesh Seetharam)

Release Version: 0.5-incubating
  INCOMPATIBLE CHANGES
    FALCON-11 Add support for security in Falcon (Venkatesh Seetharam)

    FALCON-288 Persist lineage information into a persistent store
    (Venkatesh Seetharam)

  NEW FEATURES
    FALCON-281 Design Action Interface. (Srikanth Sundarrajan)

    FALCON-259 Design Transformation Interface. (Srikanth Sundarrajan)

    FALCON-26 Pipeline Monitor addition. (Suhas Vasu via Shwetha GS)

    FALCON-254 Bootstrap designer module. (Srikanth Sundarrajan via Shwetha GS)

    FALCON-238 Support updates at specific time. (Shwetha GS)

    FALCON-285 Support Lineage information capture (Venkatesh Seetharam)

    FALCON-289 Provide REST APIs for discovering lineage metadata over the store.
    (Venkatesh Seetharam)

    FALCON-229 Enable SSL for Falcon REST API (Srikanth Sundarrajan via
    Venkatesh Seetharam)
   
  IMPROVEMENTS
    FALCON-453 possible LICENSE/NOTICE issues (Venkatesh Seetharam)

    FALCON-450 Add jdk 1.6 requirement in release notes (Venkatesh Seetharam)

    FALCON-449 Twiki documentation files may need ALv2 headers (Venkatesh Seetharam)

    FALCON-448 File names in the release should have apache as the prefix
    (Venkatesh Seetharam)

    FALCON-439 Update the copyright in Notice and License files (Venkatesh Seetharam)

    FALCON-324 Document lineage feature (Sowmya Ramesh via Venkatesh Seetharam)

    FALCON-312 Falcon LogCleanupServiceTest seems to clean up root "/"
    (Venkatesh Seetharam)

    FALCON-379 Document limitations of Hive support in Falcon (Venkatesh Seetharam)

    FALCON-421 Upgrade Hadoop to 2.4.0 (Sowmya Ramesh via Venkatesh Seetharam)

    FALCON-386 Upgrade to oozie 4.0.1. (Sowmya Ramesh via Venkatesh Seetharam)

    FALCON-392 HCat Example Recipes. (Shwetha GS via Venkatesh Seetharam)

    FALCON-391 Add ability to set mapBandwidth. (Michael Miklavcic via Shwetha GS)

    FALCON-380 The dependency option doesn't mention input or output for a feed.
    (Suhas Vasu via Shwetha GS)

    FALCON-365 Remove the checked in oozie xsds. (Shwetha GS)

    FALCON-356 Merge OozieProcessMapper and OozieProcessWorkflowBuilder. (Shwetha GS)

    FALCON-355 Remove SLAMonitoringService. (Shwetha GS)

    FALCON-333 jsp-api dependency is defined twice. (Jean-Baptiste
    Onofrévia Shaik Idris)

    FALCON-311 Several dead links in Falcon documentation.
    (Suresh Srinivas via Venkatesh Seetharam)

    FALCON-304 Simplify assembly for script in standalone and distributed
    mode. (Suresh Srinivas via Venkatesh Seetharam)

    FALCON-264 Update with effectiveTime in CLI. (Shwetha GS via Srikanth
    Sundarrajan)

    FALCON-60 Feed retention doesn't delete empty parent dirs. (Shaik Idris)

    FALCON-247 Add scripts to check the status of falcon and prism. (Jean-Baptiste 
    Onofré via Shwetha GS)

    FALCON-245 POM should use Apache POM as parent. (Jean-Baptiste Onofré via 
    Shwetha GS)

    FALCON-252 Upgrade to json-simple 1.1.1. (Jean-Baptiste Onofré via 
    Shwetha GS)

    FALCON-233 Update hadoop 2 version to 2.2.0. (Venkatesh Seetharam
    via Shwetha GS)

    FALCON-64 Add example entity xmls in falcon package. (Shwetha GS)
    
    FALCON-66 Make oozie version change configurable. (Shwetha GS
    via Srikanth Sundarrajan)
    
    FALCON-38 Falcon's parent workflow actions (pre-processing & prost-processing) 
    should have multiple retries. (Shaik Idris)

    FALCON-242 Post processing is not called in Retention workflows (Shaik Idris)

    FALCON-286 Capture information in process entity about the user workflow
    (Venkatesh Seetharam)

    FALCON-287 Record lineage information in post processing (Venkatesh Seetharam)

    FALCON-257 File system storage wont work with relative paths
    (Venkatesh Seetharam)

    FALCON-334 Add indexing to the graph property keys. (Venkatesh Seetharam)

    FALCON-342 Add ability to set maxMaps on replication (Michael Miklavcic
    via Venkatesh Seetharam)

    FALCON-366 Add a REST API to get properties for a given vertex (Venkatesh Seetharam)

    FALCON-370 Remove Graph dump option in CLI (Venkatesh Seetharam)

    FALCON-367 Bump dagre and jquery version for the web UI (Haohui Mai
    via Venkatesh Seetharam)

    FALCON-290 Visualize lineage information on the dashboard (Haohui Mai
    via Venkatesh Seetharam)

    FALCON-371 Show vertex information in the web UI (Haohui Mai via Venkatesh Seetharam)

    FALCON-373 Display lineage link only for jobs that are succeeded in the web UI
    (Haohui Mai via Venkatesh Seetharam)

    FALCON-393 Display error messages when the web UI fails to get the data from the server
    (Haohui Mai via Venkatesh Seetharam)

    FALCON-395 Falcon service does not start when _HOST is used in the spnego principal
    (Venkatesh Seetharam)

    FALCON-397 DistCp uses JobSubmissionFiles API which is incompatible between hadoop versions
    (Venkatesh Seetharam)

  OPTIMIZATIONS
    FALCON-123 Improve build speeds in falcon. (Srikanth Sundarrajan via Shwetha GS)

  BUG FIXES
    FALCON-447 Remove the jar file form the source distribution (Venkatesh Seetharam)

    FALCON-440 Exclude IDEA IntelliJ and other unnecessary files from source
    distribution (Venkatesh Seetharam)

    FALCON-437 Feed Replication workflows are failing (Venkatesh Seetharam)

    FALCON-429 Falcon CLI entity list command throws NPE stack trace in case
    of no entities (Satish Mittal via Venkatesh Seetharam)

    FALCON-427 Update default falcon endpoint URL in runtime.properties for
    prism setup to work out of box (Satish Mittal via Venkatesh Seetharam)

    FALCON-425 src/bin/package.sh is broken for hadoop 2.* versions
    (Sowmya Ramesh via Venkatesh Seetharam)

    FALCON-426 Prism unable to connect to falcon server if falcon endpoint URL ends with
    '/' character. (Satish Mittal via Shwetha GS)

    FALCON-401 replicate hcat table to two targets can fail with export dir
    exists on source (Sowmya Ramesh via Venkatesh Seetharam)

    FALCON-364 Process specification contains a mistake in the parallel section
    (Jean-Baptiste Onofré via Venkatesh Seetharam)

    FALCON-398 Null pointer Exception on submitting HCat Feed (Sowmya Ramesh
    via Venkatesh Seetharam)

    FALCON-390 falcon HCatProcess tests are failing in secure clusters
    (Venkatesh Seetharam)

    FALCON-284 Hcatalog based feed retention doesn't work when partition filter spans across
    multiple partition keys. (Satish Mittal via Shwetha GS)

    FALCON-409 Not able to create a package. (Raju Bairishetti via Shwetha GS)

    FALCON-396 minor logging typo in FalconTopicSubscriber. (Raghav Kumar Gautam via Shwetha GS)

    FALCON-352 old properties not removed on refresh of runtime properties. (Shaik Idris)

    FALCON-332 Oozie execution order is wrongly mapped in Falcon for LAST_ONLY. 
    (Shaik Idris)

    FALCON-374 Update with effective time in CLI. (Shwetha GS)

    FALCON-98 starting embedded hadoop fails sometimes. (Shwetha GS)

    FALCON-28 unable to submit/delete feed or process which had been attempted submit 
    with wrong cluster. (Shwetha GS)

    FALCON-269 Mistake in the embedded ActiveMQ port property. (Jean-Baptiste Onofré
    via Shwetha GS)

    FALCON-344 FeedEvictorTest. (Shaik Idris Ali via Shwetha GS)

    FALCON-302 Issues with distributed package properties. (Shwetha GS)

    FALCON-331 Summary API returning NPE if only start is passed. (Shwetha GS)

    FALCON-323 Summary API deserialization fails if array has only one element.
    (Shwetha GS via Shaik Idris)

    FALCON-319 Summary instance API returning wrong exception. (Shwetha GS via 
    Shaik Idris)

    FALCON-321 Feed evictor deleting more stuff than it should. (Shaik Idris)

    FALCON-330 Build fails for the module 'Apache Falcon CLI client ' on ubuntu.
    (Deepak Marathe via Shwetha GS)

    FALCON-297 Validations on update with effective time. (Shwetha GS)

    FALCON-278 Changes in feed availability info doesn't update process. (Shwetha GS)

    FALCON-239 Build failed on build-tools due to a missing SNAPSHOT. (Srikanth 
    Sundarrajan)

    FALCON-221 Logmover is not copying all action level logs. (Srikanth Sundarrajan)

    FALCON-270 Checkstyle can not be run on a module. (Jean-Baptiste Onofré via 
    Shwetha GS)

    FALCON-260 When a process is scheduled, the user workflow is failing with 
    OozieClientException. (Shwetha GS)

    FALCON-268 Checkstyle/Findbugs issues on FalconCLI. (Jean-Baptiste Onofré via 
    Shwetha GS)

    FALCON-258 Falcon status throws an error when external jobids are missing 
    (Suhas Vasu via Shaik Idris)

    FALCON-262 Example files should use aligned dependency versions. (Jean-Baptiste Onofré
    via Shwetha GS)

    FALCON-249 Fix mistakes in docs/InstallationSteps.html. (Jean-Baptiste Onofré
    via Shwetha GS)

    FALCON-241 The job logs are not copied during Post-Processing step in Replication workflows.
    (Suhas Vasu via Shwetha GS)

    FALCON-106 Falcon compilation fails for hadoop-2 profile. (Shwetha GS
    via Srikanth Sundarrajan)

    FALCON-206 Process update for wf changes. (Shwetha GS)

    FALCON-236 Falcon process output events which is optional should have at least one event 
    if defined. (Shaik Idris Ali via Shwetha GS)

    FALCON-228 Build failed on build-tools module. (Jean-Baptiste Onofré
    via Shwetha GS)

    FALCON-232 mvn clean verify fails with oozie war not found. (Shwetha GS)

    FALCON-124 unable to schedule deleted feed. (Shwetha GS via Shaik Idris)

    FALCON-226 Tests for subscribing/listening to multiple active-mq topics.  (Shaik Idris 
    via Shwetha GS)

    FALCON-336 Fix check style error - builds fail (Venkatesh Seetharam)

    FALCON-354 Bug when MetadataMappingService is not configured as one of the
    application services. (Venkatesh Seetharam)

    FALCON-360 Lineage recording fails with NPE for processes with >1 inputs.
    (Venkatesh Seetharam)

    FALCON-378 Feed status fails with NPE post update when the new coordinator
    is in PREP state (Venkatesh Seetharam)

    FALCON-377 Unit test fails consistently - EntityManagerJerseyIT.testProcessInputUpdate.
    (Shwetha GS via Venkatesh Seetharam)

    FALCON-418 FALCON-395 introduces a regression with hadoop-1
    (Jean-Baptiste Onofré via Venkatesh Seetharam)

Release Version: 0.4-incubating

   NEW FEATURES

    FALCON-54: Falcon support for hadoop 2.0

    FALCON-85: Hive (HCatalog) integration

    FALCON-164: Provide Falcon Prism Web UI

    FALCON-107: Adding extensions

   IMPROVEMENTS

    FALCON-73: Remove JUnit dependency as its used only for asserts while TestNG for tests

    FALCON-80: Option in falcon-start to not start embedded activemq

    FALCON-83: Option to build distributed packages

    FALCON-131: Remove set -e from falcon scripts

    FALCON-151: Document the REST APIs for Falcon

    FALCON-165: simplify packaging to create one package instead of client, falcon and prism packages

    FALCON-192: Release falcon 0.4 umbrella

    FALCON-62: Falcon compilation with hadoop 2.0 libs

    FALCON-63: Ability to ingest hadoop libs to falcon release package

    FALCON-67: Remove ant dependency in oozie workflows

    FALCON-76: Use cluster readonly endpoint for replication source

    FALCON-78: Falcon error when prism on one hadoop version and server on another

    FALCON-86: Hive table integration with cluster entity

    FALCON-87: Hive table integration with feed entity

    FALCON-88: Add embedded hive and webhcat for integration tests

    FALCON-90: Propagate hive table through Process mapper

    FALCON-91: Handle feed updates in purview of tables

    FALCON-93: Replication to handle hive table replication

    FALCON-94: Retention to handle hive table eviction

    FALCON-95: Enable embedding hive scripts directly in a process

    FALCON-96: Hive client to talk to the metastore

    FALCON-102: Add integration tests for feed entity parser with table defined

    FALCON-103: Upgrade oozie to 4.0.x

    FALCON-104: FALCON-86 introduces a backward incompatible change

    FALCON-113: Update documentation for Hive integration

    FALCON-129: Disable Late data handling for hive tables

    FALCON-143: Enable Late data handling for hive tables

    FALCON-146: Javadoc generation should respect skipCheck property

    FALCON-153: Feed Storage type isn't passed in replication for successful post processing

    FALCON-154: Remove webhcat startup in HadoopStartupListener as its not used

    FALCON-161: Feed evictor evicts instances not eligible when pattern has dash

    FALCON-163: Merge FALCON-85 branch into main line

    FALCON-171: Provide status in /api/entities/list

    FALCON-175: Visualize dependency information

    FALCON-178: Implement client-side pagination

    FALCON-190: /api/entities/list should allow the client to query different fields

    FALCON-193: Update the documentation to reflect the current work of dashboard

    FALCON-43: Add Falcon to Sonar for Analysis

    FALCON-134: Remove jsch dependency

    FALCON-135: Remove jzlib dependency

    FALCON-136: Upgrade to commons-codec 1.8

    FALCON-137: Remove commons-logging dependency

    FALCON-138: Remove perf4j dependency

    FALCON-198 Update LICENSE.txt to contain license information for all third-party libraries

    FALCON-363 Jetty is started with war when the war is already expanded
    (Venkatesh Seetharam)

    FALCON-358 REST API does not conform to Rexster (Venkatesh Seetharam)

    FALCON-359 Instance id's captured are of different formats in process and feed
    (Venkatesh Seetharam)

  BUG FIXES

    FALCON-223: hive-exec bundles protobuf-2.4.1 which is incompatible with hadoop-2 requiring protobuf-2.5

    FALCON-213: Validating Storage Exists (table) should be called only from the server (not from prism)

    FALCON-24: Allow re-run of timedout instances

    FALCON-42: Simplify Build and Installation Guide

    FALCON-50: Falcon replication having issue to s3

    FALCON-61: The identifier regex for oozie bundle,coordinator, workflow pattern is incorrect.

    FALCON-69: exception from ConfigurationChangeListener should fail the API

    FALCON-71: mvn jetty:run doesn't work

    FALCON-72: Feeds with invalid oozie URI in cluster cannot be deleted

    FALCON-74: Falcon CLI needs to be more robust in handling trailing spaces

    FALCON-79: Fix typos in Falcon architecture document

    FALCON-100: Build error with mvn 3.0.4

    FALCON-101: mvn release:prepare fails because of build order

    FALCON-127: Fix test issues after FALCON-107

    FALCON-128: Feed replication post processing log mover error

    FALCON-130: Use Geronimo Specs JMS instead of javax.jms

    FALCON-132: assembly plugin for embedded does not generate client artifacts

    FALCON-142: Cleanup testng dependency

    FALCON-152: REST API for entity & Admin resources only returns XML

    FALCON-159: Version API endpoint does not emit valid JSON

    FALCON-162: Update falcon scripts to let the user set the log and pid locations.

    FALCON-168: Different validity of clusters in feed not handled

    FALCON-169: multiple "/" in target for replication for multi target feed

    FALCON-174: bind activemq to all interfaces rather than just localhost

    FALCON-179: Table replication must drop partition before import as late reruns fails

    FALCON-180: Disable table replication for multiple sources

    FALCON-182: Disable specifying partitions in inputs with table storage for process

    FALCON-185: Timestamp handled as int

    FALCON-186: Enable scheduler support for embedded activemq

    FALCON-187: Update Falcon start script to also redirect stdout to the log file

    FALCON-188: hadoop-2 profile is not carried through to oozie package

    FALCON-194 Import external libraries into the repository

    FALCON-204 Process mapper generates params assuming table input and output
    as names literally (Venkatesh Seetharam)

    FALCON-201 Package oozie-el-extensions jar files as part of tarball
    (Ashish Singh via Venkatesh Seetharam)

    FALCON-203 Do not generate transitive closure for the dependency graph
    (Haohui Mai via Venkatesh Seetharam)

    FALCON-212 OozieClient does not propagate user in request to
    Oozie server (Venkatesh Seetharam)

    FALCON-208 Improve the icons for instance lists. (Haohui Mai)

    FALCON-158 Fix failing jenkins nightly build. (Venkatesh Seetharam)

    FALCON-215 Falcon-start fails. (Arpit Gupta via Venkatesh Seetharam)



Release Version: 0.3-incubating

    FALCON-70 Confirm if v0.3 branch is ready for release as per guideline.
    (Venkatesh Seetharam via Srikanth Sundarrajan)

    FALCON-65 Avoid bundling compilation path dependent configs in artifact
    and allow for config scan to seamlessly fallback to classpath if not found
    in default conf location. (Srikanth Sundarrajan via Venkatesh Seetharam)

    FALCON-58 Gaps in release source / binary tarball. (Srikanth Sundarrajan)

    FALCON-57 Build fails if the source is not in scm. (Srikanth Sundarrajan)

    FALCON-53 Add -incubating to the version. (Jarek Jarcec Cecho via Srikanth
    Sundarrajan)

    FALCON-56 Update Falcon packing to include all source files. (Srikanth
    Sundarrajan)

    FALCON-55 Update suspends old oozie coords. (Shwetha GS via Srikanth 
    Sundarrajan)

    FALCON-52 Main module configured used with jetty:run has issues with app 
    start. (Srikanth Sundarrajan)

    FALCON-30 Enable embedding pig scripts directly in a process. (Venkatesh
    Seetharam via Srikanth Sundarrajan)

    FALCON-47 Falcon Replication should support configurable delays in feed, 
    parallel, timeout and bulk transfer with variable frequency (Shaik Idris
    Ali via Srikanth Sundarrajan)

    FALCON-49 Dependency on oozie-client pulls in hadoop-auth-0.23.1 which 
    is not necessary (Venkatesh Seetharam via Srikanth Sundarrajan)

    FALCON-32 Store test output in file rather then printing all the output
    into the console (Srikanth Sundarrajan via Venkatesh Seetharam)

    FALCON-1 Create packaging and scripts to install and try Apache Falcon
    (Srikanth Sundarrajan via Venkatesh Seetharam)

    FALCON-29 Add ability to tag/classify data sets and processes to enable
    discovery (Venkatesh Seetharam via Srikanth Sundarrajan)

    FALCON-31 File Installation-steps.txt contains old path to repository (Jarek 
    Jarcec Cecho via Srikanth Sundarrajan)

    FALCON-33 Add patch files to RAT ignore list (Jarek Jarcec Cecho via Srikanth
    Sundarrajan)

    FALCON-22 Rename ivory to falcon in project (Venkatesh Seetharam via Srikanth
    Sundarrajan)

    FALCON-19 Bump up the release version to 0.3-SNAPSHOT (Srikanth Sundarrajan 
    via Venkatesh Seetharam)

    FALCON-13 super pom must reflect license, project website, ML and SCM changes 
    post incubation (Venkatesh Seetharam via Srikanth Sundarrajan) 

    FALCON-10 Add findbugs plugin and fix findbugs warnings for project (Venkatesh
    Seetharam via Srikanth Sundarrajan)