-
Notifications
You must be signed in to change notification settings - Fork 3
/
Copy pathCHANGES.txt
3818 lines (2073 loc) · 136 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
Pig Change Log
Trunk (unreleased changes)
INCOMPATIBLE CHANGES
PIG-3191: [piggybank] MultiStorage output filenames are not sortable (Danny Antonelli via jcoveney)
PIG-3174: Remove rpm and deb artifacts from build.xml (gates)
IMPROVEMENTS
PIG-3015: Rewrite of AvroStorage (jadler via cheolsoo)
PIG-3361: Improve Hadoop version detection logic for Pig unit test (daijy)
PIG-3280: Document IN operator and CASE expression (cheolsoo)
PIG-3342: Allow conditions in case statement (cheolsoo)
PIG-3327: Pig hits OOM when fetching task reports (rohini)
PIG-3336: Change IN operator to use or-expressions instead of EvalFunc (cheolsoo)
PIG-3339: Move pattern compilation in ToDate as a static variable (rohini)
PIG-3332: Upgrade Avro dependency to 1.7.4 (nielsbasjes via cheolsoo)
PIG-3307: Refactor physical operators to remove methods parameters that are always null (julien)
PIG-3317: disable optimizations via pig properties (traviscrawford via billgraham)
PIG-3321: AVRO: Support user specified schema on load (harveyc via rohini)
PIG-2959: Add a pig.cmd for Pig to run under Windows (daijy)
PIG-3311: add pig-withouthadoop-h2 to mvn-jar (julien)
PIG-2873: Converting bin/pig shell script to python (vikram.dixit via daijy)
PIG-3308: Storing data in hive columnar rc format (maczech via daijy)
PIG-3303: add hadoop h2 artifact to publications in ivy.xml (julien)
PIG-3169: Remove intermediate data after a job finishes (mwagner via cheolsoo)
PIG-3173: Partition filter push down does not happen when partition keys condition include a AND and OR construct (rohini)
PIG-2786: enhance Pig launcher script wrt. HBase/HCat integration (ndimiduk via daijy)
PIG-3198: Let users use any function from PigType -> PigType as if it were builtlin (jcoveney)
PIG-3268: Case statement support (cheolsoo)
PIG-3269: In operator support (cheolsoo)
PIG-200: Pig Performance Benchmarks (daijy)
PIG-3261: User set PIG_CLASSPATH entries must be prepended to the CLASSPATH, not
appended (qwertymaniac via daijy)
PIG-3141: Giving CSVExcelStorage an option to handle header rows (jpacker via cheolsoo)
PIG-3217: Add support for DateTime type in Groovy UDFs (herberts via daijy)
PIG-3218: Add support for biginteger/bigdecimal type in Groovy UDFs (herberts via daijy)
PIG-3248: Upgrade hadoop-2.0.0-alpha to hadoop-2.0.3-alpha (daijy)
PIG-3235: Add log4j.properties for unit tests (cheolsoo)
PIG-3236: parametrize snapshot and staging repo id (gkesavan via daijy)
PIG-3244: Make PIG_HOME configurable ([email protected] via daijy)
PIG-3233: Deploy a Piggybank Jar (njw45 via cheolsoo)
PIG-3245: Documentation about HBaseStorage (Daisuke Kobayashi via cheolsoo)
PIG-3211: Allow default Load/Store funcs to be configurable (prkommireddi via cheolsoo)
PIG-3136: Introduce a syntax making declared aliases optional (jcoveney via cheolsoo)
PIG-3142: [piggybank] Fixed-width load and store functions for the Piggybank (jpacker via cheolsoo)
PIG-3162: PigTest.assertOutput doesn't allow non-default delimiter (dreambird via cheolsoo)
PIG-3002: Pig client should handle CountersExceededException (jarcec via billgraham)
PIG-3189: Remove ivy/pig.pom and improve build mvn targets (billgraham)
PIG-3192: Better call to action to download Pig in docs (rjurney via jcoveney)
PIG-3167: Job stats are printed incorrectly for map-only jobs (Mark Wagner via jcoveney)
PIG-3131: Document PluckTuple UDF (rjurney via jcoveney)
PIG-3098: Add another test for the self join case (jcoveney)
PIG-3129: Document syntax to refer to previous relation (rjurney via jcoveney)
PIG-2553: Pig shouldn't allow attempts to write multiple relations into same directory (prkommireddi via cheolsoo)
PIG-3179: Task Information Header only prints out the first split for each task (knoguchi via rohini)
PIG-3108: HBaseStorage returns empty maps when mixing wildcard with other columns (christoph.bauer via billgraham)
PIG-3178: Print a stacktrace when ExecutableManager hits an OOM (knoguchi via rohini)
PIG-3160: GFCross uses unnecessary loop (sandyr via cheolsoo)
PIG-3138: Decouple PigServer.executeBatch() from compilation of batch (pkommireddi via cheolsoo)
PIG-2878: Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This
check is case insensitive. (shami via gates)
PIG-2994: Grunt shortcuts (prasanth_j via cheolsoo)
PIG-3140: Document PigProgressNotificationListener configs (billgraham)
PIG-3139: Document reducer estimation (billgraham)
PIG-2764: Add a biginteger and bigdecimal type to pig (jcoveney)
PIG-3073: POUserFunc creating log spam for large scripts (jcoveney)
PIG-3082: outputSchema of a UDF allows two usages when describing a Tuple schema (jcoveney)
PIG-3124: Push FLATTENs After FILTERs If Possible (nwhite via daijy)
PIG-3086: Allow A Prefix To Be Added To URIs In PigUnit Tests (nwhite via gates)
PIG-3091: Make schema, header and stats file configurable in JsonMetadata (pkommireddi via jcoveney)
PIG-3078: Make a UDF that, given a string, returns just the columns prefixed by that string (jcoveney)
PIG-3090: Introduce a syntax to be able to easily refer to the previously defined relation (jcoveney)
PIG-3057: Make PigStorage.readField() protected (pablomar and billgraham via billgraham)
PIG-2788: improved string interpolation of variables (jcoveney)
PIG-2362: Rework Ant build.xml to use macrodef instead of antcall (azaroth via cheolsoo)
PIG-2857: Add a -tagPath option to PigStorage (prkommireddi via cheolsoo)
PIG-2341: Need better documentation on Pig/HBase integration (jthakrar and billgraham via billgraham)
PIG-3075: Allow AvroStorage STORE Operations To Use Schema Specified By URI (nwhite via cheolsoo)
PIG-3062: Change HBaseStorage to permit overriding pushProjection (billgraham)
PIG-3016: Modernize more tests (jcoveney via cheolsoo)
PIG-2582: Store size in bytes (not mbytes) in ResourceStatistics (prkommireddi via billgraham)
PIG-3006: Modernize a chunk of the tests (jcoveney via cheolsoo)
PIG-2997: Provide a convenience constructor on PigServer that accepts Configuration (prkommireddi via rohini)
PIG-2933: HBaseStorage is using setScannerCaching which is deprecated (prkommireddi via rohini)
PIG-2881: Add SUBTRACT eval function (jocosti via cheolsoo)
PIG-3004: Improve exceptions messages when a RuntimeException is raised in Physical Operators (julien)
PIG-2990: the -secretDebugCmd shouldn't be a secret and should just be...a command (jcoveney)
PIG-2941: Ivy resolvers in pig don't have consistent chaining and don't have a kitchen sink option for novices (jgordon via azaroth)
PIG-2778: Add 'matches' operator to predicate pushdown (cheolsoo via jcoveney)
PIG-2966: Test failures on CentOS 6 because MALLOC_ARENA_MAX is not set (cheolsoo via sms)
PIG-2794: Pig test: add utils to simplify testing on Windows (jgordon via gates)
PIG-2910: Add function to read schema from outout of Schema.toString() (initialcontext via thejas)
OPTIMIZATIONS
PIG-3013: BinInterSedes improve chararray sort performance (rohini)
BUG FIXES
PIG-3364: Case expression fails with an even number of when branches (cheolsoo)
PIG-3354: UDF example does not handle nulls (patc888 via daijy)
PIG-3355: ColumnMapKeyPrune bug with distinct operator (jeremykarn via aniket486)
PIG-3318: AVRO: 'default value' not honored when merging schemas on load with AvroStorage (viraj via rohini)
PIG-3250: Pig dryrun generates wrong output in .expanded file for 'SPLIT....OTHERWISE...' command (dreambird via cheolsoo)
PIG-3331: Default values not stored in avro file when using specific schemas during store in AvroStorage (viraj via rohini)
PIG-3322: AvroStorage give NPE on reading file with union as top level schema (viraj via rohini)
PIG-2828: Handle nulls in DataType.compare (aniket486)
PIG-3335: TestErrorHandling.tesNegative7 fails on MR2 (xuefuz)
PIG-3316: Pig failed to interpret DateTime values in some special cases (xuefuz)
PIG-2956: Invalid cache specification for some streaming statement (daijy)
PIG-3310: ImplicitSplitInserter does not generate new uids for nested schema fields, leading to miscomputations (cstenac via daijy)
PIG-3334: Fix Windows piggybank unit test failures (daijy)
PIG-3337: Fix remaining Window e2e tests (daijy)
PIG-3328: DataBags created with an initial list of tuples don't get registered as spillable (mwagner via daijy)
PIG-3313: pig job hang if the job tracker is bounced during execution (yu.chenjie via daijy)
PIG-3297: Avro files with stringType set to String cannot be read by the AvroStorage LoadFunc (nielsbasjes via cheolsoo)
PIG-3069: Native Windows Compatibility for Pig E2E Tests and Harness (anthony.murphy via daijy)
PIG-3291: TestExampleGenerator fails on Windows because of lack of file name escaping (dwann via daijy)
PIG-3026: Pig checked-in baseline comparisons need a pre-filter to address OS-specific newline differences (dwann via daijy)
PIG-3025: TestPruneColumn unit test - SimpleEchoStreamingCommand perl inline script needs simplification (dwann via daijy)
PIG-2955: Fix bunch of Pig e2e tests on Windows (daijy)
PIG:3302: JSONStorage throws NPE if map has null values (rohini)
PIG-3309: TestJsonLoaderStorage fails with IBM JDK 6/7 (lrangel via daijy)
PIG-3097: HiveColumnarLoader doesn't correctly load partitioned Hive table (maczech via daijy)
PIG-3305: Infinite loop when input path contains empty partition directory (maczech via daijy)
PIG-3286: TestPigContext.testImportList fails in trunk (cheolsoo)
PIG-2970: Nested foreach getting incorrect schema when having unrelated inner query (daijy)
PIG-3304: XMLLoader in piggybank does not work with inline closed tags (aseldawy via daijy)
PIG-3028: testGrunt dev test needs some command filters to run correctly without cygwin (jgordon via gates)
PIG-3290: TestLogicalPlanBuilder.testQuery85 fail in trunk (daijy)
PIG-3027: pigTest unit test needs a newline filter for comparisons of golden multi-line (jgordon via gates)
PIG-2767: Pig creates wrong schema after dereferencing nested tuple fields (daijy)
PIG-3276: change the default value for hcat.bin to hcat instead of /usr/local/hcat/bin/hcat (arpitgupta via daijy)
PIG-3277: fix the path to the benchmarks file in the print statement (arpitgupta via daijy)
PIG-3122: Operators should not implicitly become reserved keywords (jcoveney via cheolsoo)
PIG-3193: Fix "ant docs" warnings (cheolsoo)
PIG-3186: tar/deb/pkg ant targets should depend on piggybank (lbendig via gates)
PIG-3270: Union onschema failing at runtime when merging incompatible types (knoguchi via daijy)
PIG-3271: POSplit ignoring error from input processing giving empty results (knoguchi via daijy)
PIG-2265: Test case TestSecondarySort failure (daijy)
PIG-3060: FLATTEN in nested foreach fails when the input contains an empty bag (daijy)
PIG-2247: Pig parser does not detect multiple arguments with the same name passed to macro (dreambird via daijy)
PIG-3249: Pig startup script prints out a wrong version of hadoop when using fat jar (prkommireddi via daijy)
PIG-3110: pig corrupts chararrays with trailing whitespace when converting them to long (prkommireddi via daijy)
PIG-3253: Misleading comment w.r.t getSplitIndex() method in PigSplit.java (cheolsoo)
PIG-3208: [zebra] TFile should not set io.compression.codec.lzo.buffersize (ekoontz via daijy)
PIG-3172: Partition filter push down does not happen when there is a non partition key map column filter (rohini)
PIG-3205: Passing arguments to python script does not work with -f option (rohini)
PIG-2507: Semicolon in paramenters for UDF results in parsing error (tnachen via daijy)
PIG-3239: Unable to return multiple values from a macro using SPLIT (dreambird via cheolsoo)
PIG-3077: TestMultiQueryLocal should not write in /tmp (dreambird via cheolsoo)
PIG-3081: Pig progress stays at 0% for the first job in hadoop 23 (rohini)
PIG-3150: e2e Scripting_5 fails in trunk (dreambird via cheolsoo)
PIG-3153: TestScriptUDF.testJavascriptExampleScript fails in trunk (cheolsoo)
PIG-3145: Parameters in core-site.xml and mapred-site.xml are not correctly substituted (cheolsoo)
PIG-3135: HExecutionEngine should look for resources in user passed Properties (prkommireddi via cheolsoo)
PIG-3200: MiniCluster should delete hadoop-site.xml on shutDown (prkommireddi via cheolsoo)
PIG-3158: Errors in the document "Control Structures" (miyakawataku via cheolsoo)
PIG-3161: Update reserved keywords in Pig docs (russell.jurney via cheolsoo)
PIG-3156: TestSchemaTuple fails in trunk (cheolsoo)
PIG-3155: TestTypeCheckingValidatorNewLP.testSortWithInnerPlan3 fails in trunk (cheolsoo)
PIG-3154: TestPackage.testOperator fails in trunk (dreambird via cheolsoo)
PIG-3168: TestMultiQueryBasic.testMultiQueryWithSplitInMapAndMultiMerge fails in trunk (cheolsoo)
PIG-3137: Fix Piggybank test to not using /tmp dir (dreambird via cheolsoo)
PIG-3149: e2e build.xml still refers to jython 2.5.0 jar even though it's replaced by jython standalone 2.5.2 jar (cheolsoo)
PIG-2266: Bug with input file joining optimization in Pig (jadler via cheolsoo)
PIG-2645: PigSplit does not handle the case where SerializationFactory returns null (shami via gates)
PIG-3031: Update Pig to use a newer version of joda-time (zjshen via cheolsoo)
PIG-3071: Update hcatalog jar and path to hbase storage handler jar in pig script (arpitgupta via cheolsoo)
PIG-3029 TestTypeCheckingValidatorNewLP has some path reference issues for cross-platform execution (jgordon via gates)
PIG-3120: setStoreFuncUDFContextSignature called with null signature (jdler via cheolsoo)
PIG-3115: Distinct Build-in Function Doesn't Handle Null Bags (njw45 via daijy)
PIG-2433: Jython import module not working if module path is in classpath (rohini)
PIG-2769 a simple logic causes very long compiling time on pig 0.10.0 (nwhite via gates)
PIG-2251: PIG leaks Zookeeper connections when using HBaseStorage (jamarkha via cheolsoo)
PIG-3112: Errors and lacks in document "User Defined Functions" (miyakawataku via cheolsoo)
PIG-3050: Fix FindBugs multithreading warnings (cheolsoo)
PIG-3066: Fix TestPigRunner in trunk (cheolsoo)
PIG-3101: Increase io.sort.mb in YARN MiniCluste (cheolsoo)
PIG-3100: If a .pig_schema file is present, can get an index out of bounds error (jcoveney)
PIG-3096: Make PigUnit thread safe (cheolsoo)
PIG-3095: "which" is called many, many times for each Pig STREAM statement (nwhite via cheolsoo)
PIG-3085: Errors and lacks in document "Built In Functions" (miyakawataku via cheolsoo)
PIG-3084: Improve exceptions messages in POPackage (julien)
PIG-3072: Pig job reporting negative progress (knoguchi via rohini)
PIG-3014: CurrentTime() UDF has undesirable characteristics (jcoveney via cheolsoo)
PIG-2924: PigStats should not be assuming all Storage classes to be file-based storage (cheolsoo)
PIG-3046: An empty file name in -Dpig.additional.jars throws an error (prkommireddi via cheolsoo)
PIG-2989: Illustrate for Rank Operator (xalan via gates)
PIG-2885: TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3 (cheolsoo via sms)
PIG-2928: Fix e2e test failures in trunk: FilterBoolean_23/24 (cheolsoo via dvryaboy)
Release 0.11.2 (Unreleased)
INCOMPATIBLE CHANGES
IMPROVEMENTS
OPTIMIZATIONS
PIG-2769: a simple logic causes very long compiling time on pig 0.10.0 (njw45 via dvryaboy) (prev. applied to 0.12)
BUG FIXES
PIG-3341: Strict datetime parsing and improve performance of loading datetime values (rohini)
PIG-3329: RANK operator failed when working with SPLIT (xalan via cheolsoo)
PIG-3345: Handle null in DateTime functions (rohini)
PIG-3223: AvroStorage does not handle comma separated input paths (dreambird via rohini)
PIG-3262: Pig contrib 0.11 doesn't compile on certain rpm systems (mgrover via cheolsoo)
PIG-3264: mvn signanddeploy target broken for pigunit, pigsmoke and piggybank (billgraham)
Release 0.11.1
INCOMPATIBLE CHANGES
IMPROVEMENTS
PIG-3256: Upgrade jython to 2.5.3 (legal concern) (daijy)
PIG-2988: start deploying pigunit maven artifact part of Pig release process (njw45 via rohini)
PIG-3148: OutOfMemory exception while spilling stale DefaultDataBag. Extra option to gc() before spilling large bag. (knoguchi via rohini)
PIG-3216: Groovy UDFs documentation has minor typos (herberts via rohini)
PIG-3202: CUBE operator not documented in user docs (prasanth_j via billgraham)
OPTIMIZATIONS
BUG FIXES
PIG-3267: HCatStorer fail in limit query (daijy)
PIG-3252: AvroStorage gives wrong schema for schemas with named records (mwagner via cheolsoo)
PIG-3132: NPE when illustrating a relation with HCatLoader (daijy)
PIG-3194: Changes to ObjectSerializer.java break compatibility with Hadoop 0.20.2 (prkommireddi via dvryaboy)
PIG-3241: ConcurrentModificationException in POPartialAgg (dvryaboy)
PIG-3144: Erroneous map entry alias resolution leading to "Duplicate schema alias" errors (jcoveney via cheolsoo)
PIGG-3212: Race Conditions in POSort and (Internal)SortedBag during Proactive Spill (kadeng via dvryaboy)
PIG-3206: HBaseStorage does not work with Oozie pig action and secure HBase (rohini)
Release 0.11.0
INCOMPATIBLE CHANGES
PIG-3034: Remove Penny code from Pig repository (gates via cheolsoo)
PIG-2931: $ signs in the replacement string make parameter substitution fail (cheolsoo via jcoveney)
PIG-1891 Enable StoreFunc to make intelligent decision based on job success or failure (initialcontext via gates)
IMPROVEMENTS
PIG-3044: Trigger POPartialAgg compaction under GC pressure (dvryaboy)
PIG-2907: Publish pig jars for Hadoop2/23 to maven (rohini)
PIG-2934: HBaseStorage filter optimizations (billgraham)
PIG-2980: documentation for DateTime datatype (zjshen via thejas)
PIG-2982: add unit tests for DateTime type that test setting timezone (zjshen via thejas)
PIG-2937: generated field in nested foreach does not inherit the variable name as the field name (jcoveney)
PIG-3019: Need a target in build.xml for source releases (gates)
PIG-2832: org.apache.pig.pigunit.pig.PigServer does not initialize udf.import.list of PigContext (prkommireddi via rohini)
PIG-2898: Parallel execution of e2e tests (iveselovsky via rohini)
PIG-2913: org.apache.pig.test.TestPigServerWithMacros fails sometimes because it picks up previous minicluster configuration file (cheolsoo via julien)
PIG-2976: Reduce HBaseStorage logging (billgraham)
PIG-2947: Documentation for Rank operator (xalan via azaroth)
PIG-2943: DevTests, Refactor Windows checks to use new Util.WINDOWS method for code health (jgordon via dvryaboy)
PIG-2908: Fix unit tests to work with jdk7 (rohini via dvryaboy)
PIG-2965: RANDOM should allow seed initialization for ease of testing (jcoveney)
PIG-2964: Add helper method getJobList() to PigStats.JobGraph. Extend visibility of couple methods on same class (prkommireddi via billgraham)
PIG-2579: Support for multiple input schemas in AvroStorage (cheolsoo via sms)
PIG-2946: Documentation of "history" and "clear" commands (xalan via azaroth)
PIG-2877: Make SchemaTuple work in foreach (and thus, in loads) (jcoveney)
PIG-2923: Lazily register bags with SpillableMemoryManager (dvryaboy)
PIG-2929: Improve documentation around AVG, CONCAT, MIN, MAX (cheolsoo via billgraham)
PIG-2852: Update documentation regarding parallel local mode execution (cheolsoo via jcoveney)
PIG-2879: Pig current releases lack a UDF startsWith.This UDF tests if a given string starts with the specified prefix. (initialcontext via azaroth)
PIG-2712: Pig does not call OutputCommitter.abortJob() on the underlying OutputFormat (rohini via gates)
PIG-2918: Avoid Spillable bag overhead where possible (dvryaboy)
PIG-2900: Streaming should provide conf settings in the environment (dvryaboy)
PIG-2353: RANK function like in SQL (xalan via azaroth)
PIG-2915: Builtin TOP udf is sensitive to null input bags (hazen via dvryaboy)
PIG-2901: Errors and lacks in document "Pig Latin Basics" (miyakawataku via billgraham)
PIG-2905: Improve documentation around REPLACE (cheolsoo via billgraham)
PIG-2882: Use Deque instead of Stack (mkhadikov via dvryaboy)
PIG-2781: LOSort isEqual method (xalan via dvryaboy)
PIG-2835: Optimizing the convertion from bytes to Integer/Long (jay23jack via dvryaboy)
PIG-2886: Add Scan TimeRange to HBaseStorage (ted.m via dvryaboy)
PIG-2895: jodatime jar missing in pig-withouthadoop.jar (thejas)
PIG-2888: Improve performance of POPartialAgg (dvryaboy)
PIG-2708: split MiniCluster based tests out of org.apache.pig.test.TestInputOutputFileValidator (analog.sony via daijy)
PIG-2890: Revert PIG-2578 (dvryaboy)
PIG-2850: Pig should support loading macro files as resources stored in JAR files (matterhayes via dvryaboy)
PIG-1314: Add DateTime Support to Pig (zjshen via thejas)
PIG-2785: NoClassDefFoundError after upgrading to pig 0.10.0 from 0.9.0 (matterhayes via sms)
PIG-2556: CSVExcelStorage load: quoted field with newline as first character sees newline as record end (tivv via dvryaboy)
PIG-2875: Add recursive record support to AvroStorage (cheolsoo via sms)
PIG-2662: skew join does not honor its config parameters (rajesh.balamohan via thejas)
PIG-2871: Refactor signature for PigReducerEstimator (billgraham)
PIG-2851: Add flag to ant to run tests with a debugger port (billgraham)
PIG-2862: Hardcode certain tuple lengths into the TUPLE BinInterSedes byte identifier (jcoveney)
PIG-2855: Provide a method to measure time spent in UDFs (dvryaboy)
PIG-2837: AvroStorage throws StackOverFlowError (cheolsoo via sms)
PIG-2856: AvroStorage doesn't load files in the directories when a glob pattern matches both files and directories. (cheolsoo via sms)
PIG-2569: Fix org.apache.pig.test.TestInvoker.testSpeed (aklochkov via dvryaboy)
PIG-2858: Improve PlanHelper to allow finding any PhysicalOperator in a plan (dvryaboy)
PIG-2854: AvroStorage doesn't work with Avro 1.7.1 (cheolsoo via sms)
PIG-2779: Refactoring the code for setting number of reducers (jay23jack via billgraham)
PIG-2765: Implementing RollupDimensions UDF and adding ROLLUP clause in CUBE operator (prasanth_j via dvryaboy)
PIG-2814: Fix issues with Sample operator documentation (prasanth_j via dvryaboy)
PIG-2817: Documentation for Groovy UDFs (herberts via julien)
PIG-2492: AvroStorage should recognize globs and commas (cheolsoo via sms)
PIG-2706: Add clear to list of grunt commands (xalan via azaroth)
PIG-2823: TestPigContext.testImportList() does not pass if another javac in on the PATH (julien)
PIG-2800: pig.additional.jars path separator should align with File.pathSeparator instead of being hard-coded to ":" (jgordon via azaroth)
PIG-2797: Tests should not create their own file URIs through string concatenation, should use Util.generateURI instead (jgordon via azaroth)
PIG-2820: relToAbsolutePath is not replayed properly when Grunt reparses the script after PIG-2699 (julien)
PIG-2763: Groovy UDFs (herberts via julien)
PIG-2780: MapReduceLauncher should break early when one of the jobs throws an exception (jay23jack via daijy)
PIG-2804: Remove "PIG" exec type (dvryaboy)
PIG-2726: Handling legitimate NULL values in Cube operator (prasanth_j via dvryaboy)
PIG-2808: Add *.project to .gitignore (azaroth)
PIG-2787: change the module name in ivy to lowercase to match the maven repo (julien)
PIG-2632: Create a SchemaTuple which generates efficient Tuples via code gen (jcoveney)
PIG-2750: add artifacts to the ivy.xml for other jars Pig generates (julien)
PIG-2748: Change the names of the jar produced in the build folder to match maven conventions (julien)
PIG-2770: Allow easy inclusion of custom build targets (julien)
PIG-2697: pretty print schema via pig.pretty.print.schema (rangadi via jcoveney)
PIG-2673: Allow Merge join to follow an ORDER statement (dvryaboy)
PIG-2699: Reduce the number of instances of Load and Store Funcs down to 2+1 (julien)
PIG-2166: UDFs to join a bag (hluu via daijy)
PIG-2651: Provide a much easier to use accumulator interface (jcoveney via daijy)
PIG-2658: Add pig.script.submitted.timestamp and pig.job.submitted.timestamp in generated Map-Reduce job conf (billgraham)
PIG-2735: Add a pig.version.suffix property in build.xml to easily override with a build number (julien)
PIG-2705: outputSchema modification from scripting UDFs (levyjoshua via julien)
PIG-2724: Make Tuple Iterable (jcoveney)
PIG-2733: Add *.patch, *.log, *.orig, *.rej, *.class to gitignore (jcoveney)
PIG-2732: Let's get rid of the deprecated Tuple methods (jcoveney)
PIG-2638: Optimize BinInterSedes treatment of longs (jcoveney)
PIG-2727: PigStorage Source tagging does not need pig.splitCombination to be turned off (prkommireddi via dvryaboy)
PIG-2710: Implement Naive CUBE operator (prasanth_j via dvryaboy)
PIG-2714: Pig documentation on TOP funcation has issues (daijy)
PIG-2066: Accumulators should be able to early-terminate (jcoveney)
PIG-2600: Better Map support (prkommireddi via jcoveney)
PIG-2711: e2e harness: cache benchmark results between test runs (thw via daijy)
PIG-2702: Make Pig local mode (and tests) faster by working around the hard coded sleep(5000) in hadoop's JobControl (julien)
PIG-2659: add source location of the aliases in the physical plan (julien)
PIG-2547: Easier UDFs: Convenient EvalFunc super-classes (billgraham, dvryaboy)
PIG-2639: Utils.getSchemaFromString should automatically give name to all types, but fails on boolean (jcoveney)
PIG-2696: Enhance Job Stat to print out median map and reduce time (hluu via daijy)
PIG-2583: Add Grunt command to list the statements in cache (xalan via daijy)
PIG-2688: Log the aliases being processed for the current job (ddaniels888 via azaroth)
PIG-2680: TOBAG output schema reporting (andy schlaikjer via jcoveney)
PIG-2685: Fix error in EvalFunc ctor when implementing Algebraic UDF whose return type is parameterized (andy schlaikjer via jcoveney)
PIG-2664: Allow PPNL impls to get more job info during the run (billgraham)
PIG-2663: Expose helpful ScriptState methods (billgraham)
PIG-2660: PPNL notified of plan before it gets executed (billgraham)
PIG-2574: Make reducer estimator plugable (billgraham)
PIG-2677: Add target to build.xml to generate clover summary reports (gates)
PIG-2650: Convenience mock Loader and Storer to simplify unit testing of Pig scripts (julien)
PIG-2257: AvroStorage doesn't recognize schema_file field when JSON isn't used in the constructor (billgraham)
PIG-2587: Compute LogicalPlan signature and store in job conf (billgraham)
PIG-2619: HBaseStorage constructs a Scan with cacheBlocks = false
PIG-2604: Pig should print its build info at runtime (traviscrawford via dvryaboy)
PIG-2573: Automagically setting parallelism based on input file size does not work with HCatalog (traviscrawford via julien)
PIG-2538: Add helper wrapper classes for StoreFunc (billgraham via dvryaboy)
PIG-2010: registered jars on distributed cache (traviscrawford and julienledem via dvryaboy)
PIG-2533: Pig MR job exceptions masked on frontend (traviscrawford via dvryaboy)
PIG-2525: Support pluggable PigProcessNotifcationListeners on the command line (dvryaboy)
PIG-2515: [piggybank] Make CustomFormatToISO return null on Exception in parsing dates (rjurney via dvryaboy)
PIG-2503: Make @MonitoredUDF inherited (dvryaboy)
PIG-2488: Move Python unit tests to e2e tests (alangates via daijy)
PIG-2456: Pig should have a pigrc to specify default script cache (prkommireddi via daijy)
PIG-2496: Cache resolved classes in PigContext (dvryaboy)
PIG-2482: Integrate HCat DDL command into Pig (daijy)
PIG-2479: changingPattern should be used with checkmodified in ivysettings.xml (abayer via azaroth)
PIG-2349: Ant build repeats ivy-buildJar several times (azaroth)
PIG-2359: Support more efficient Tuples when schemas are known (dvryaboy)
PIG-2282: Automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy (azaroth via daijy)
PIG-2468: Speed up TestBuiltin (dvryaboy)
PIG-2467: Speed up TestCommit (dvryaboy)
PIG-2460: Use guava 11 instead of r06 (dvryaboy)
PIG-2267: Make the name of the columns in schema optional (jcoveney via daijy)
PIG-2453: Fetching schema can be very slow for multi-thousand file LOADs (dvryaboy)
PIG-2443: [Piggybank] Add UDFs to check if a String is an Integer And if a String is Numeric (prkommireddi via daijy)
PIG-2437: Use Ivy to get automaton.jar (azaroth)
PIG-2448: Convert more tests to use LOCAL mode (dvryaboy)
PIG-2438: Do not hardcode commons-lang version in build.xml (azaroth)
PIG-2422: Add log messages for Jython schema definitions (vivekp via gates)
PIG-2403: Reduce code duplication in SUM, MAX, MIN udfs (dvryaboy)
PIG-2245: Add end to end test for tokenize (markroddy via gates)
PIG-2327: bin/pig doesn't have any hooks for picking up ZK installation deployed from tarballs (rvs via hashutosh)
PIG-2382: Modify .gitignore to ignore pig-withouthadoop.jar (azaroth via hashutosh)
PIG-2380: Expose version information more cleanly (jcoveney via azaroth)
PIG-2311: STRSPLIT needs to allow bytearray arguments (xuting via olgan)
PIG-2365: Current TOP implementation needlessly results in a null bag name (jcoveney via dvryaboy)
PIG-2151: Add annotation to specify output schema in Java UDFs (dvryaboy)
PIG-2230: Improved error message for invalid parameter format (xuitingz via olgan)
PIG-2328: Add builtin UDFs for building and using bloom filters (gates)
PIG-2338: Need signature for EvalFunc (daijy)
PIG-2337: Provide UDF with input schema (xutingz via daijy)
OPTIMIZATIONS
BUG FIXES
PIG-3147: Spill failing with "java.lang.RuntimeException: InternalCachedBag.spill() should not be called" (knoguchi via dvryaboy)
PIG-3109: Missing license headers (jarcec via cheolsoo)
PIG-3022: TestRegisteredJarVisibility.testRegisteredJarVisibility fails with hadoop-2.0.x (rohini via cheolsoo)
PIG-3125: Fix zebra compilation error (cheolsoo)
PIG-3051: java.lang.IndexOutOfBoundsException failure with LimitOptimizer + ColumnPruning (knoguchi via rohini)
PIG-3076: make TestScalarAliases more reliable (julien)
PIG-3020: "Duplicate uid in schema" error when joining two relations derived from the same load statement (jcoveney)
PIG-3044: hotfix to remove divide by 0 error (jcoveney)
PIG-3033: test-patch failed with javadoc warnings (fang fang chen via cheolsoo)
PIG-3058: Upgrade junit to at least 4.8 (fang fang chen via cheolsoo)
PIG-2978: TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x (cheolsoo)
PIG-3039: Not possible to use custom version of jackson jars (rohini)
PIG-3045: Specifying sorting field(s) at nightly.conf - fix sortArgs (rohini via cheolsoo)
PIG-2979: Pig.jar doesn't work with hadoop-2.0.x (cheolsoo)
PIG-3035: With latest version of hadoop23 pig does not return the correct exception stack trace from backend (rohini)
PIG-2405: some unit test case failed with open JDK (fang fang chen via cheolsoo)
PIG-3018: Refactor TestScriptLanguage to remove duplication and write script in different files (julien)
PIG-2973: TestStreaming test times out (cheolsoo)
PIG-3001: TestExecutableManager.testAddJobConfToEnv fails randomly (cheolsoo)
PIG-3017: Pig's object serialization should use compression (jcoveney)
PIG-2968: ColumnMapKeyPrune fails to prune a subtree inside foreach (knoguchi via cheolsoo)
PIG-2999: Regression after PIG-2975: BinInterSedesTupleRawComparator secondary sort failing (knoguchi via azaroth)
PIG-2998: Fix TestScriptLanguage and TestMacroExpansion (cheolsoo via jcoveney)
PIG-2975: TestTypedMap.testOrderBy failing with incorrect result (knoguchi via jcoveney)
PIG-2950: Fix tiny documentation error in BagToString builtin (initialcontext via daijy)
PIG-2967: Fix Glob_local test failure for Pig E2E Test Framework (sushantj via daijy)
PIG-1283: COUNT on null bag causes failure (analog.sony via jcoveney)
PIG-2958: Pig tests do not appear to have a logger attached (daijyc via jcoveney)
PIG-2926: TestPoissonSampleLoader failing on rhel environment (jcoveney)
PIG-2985: TestRank1,2,3 fail with hadoop-2.0.x (rohini via azaroth)
PIG-2971: Add new parameter to specify the streaming environment (jcoveney)
PIG-2963: Illustrate command and POPackageLite (cheolsoo via jcoveney)
PIG-2961: BinInterSedesRawComparator broken by TUPLE_number patch (jcoveney)
PIG-2932: Setting high default_parallel causes IOException in local mode (cheolsoo via gates)
PIG-2737: [piggybank] TestIndexedStorage is failing, should be refactored (jcoveney)
PIG-2935: Catch NoSuchMethodError when StoreFuncInterface's new cleanupOnSuccess method isn't implemented. (gates via dvryaboy)
PIG-2920: e2e tests override PERL5LIB environment variable (azaroth)
PIG-2917: SpillableMemoryManager memory leak for WeakReference (haitao.yao via dvryaboy)
PIG-2938: All unit tests that use MR2 MiniCluster are broken in trunk (cheolsoo via dvryaboy)
PIG-2936: Tuple serialization bug (jcoveney)
PIG-2930: ant test doesn't compile in trunk (cheolsoo via daijy)
PIG-2791: Pig does not work with ViewFileSystem (rohini via daijy)
PIG-2833: org.apache.pig.pigunit.pig.PigServer does not initialize set default log level of pigContext (cheolsoo via jcoveney)
PIG-2744: Handle Pig command line with XML special characters (lulynn_2008 via daijy)
PIG-2637: Command-line option -e throws TokenMgrError exception (lulynn_2008 via daijy)
PIG-2887: Macro cannot handle negative number (knoguchi via gates)
PIG-2844: ant makepom is misconfigured (julien)
PIG-2896: Pig does not fail anymore if two macros are declared with the same name (julien)
PIG-2848: TestBuiltInBagToTupleOrString fails now that mock.Storage enforces not overwriting output (julien)
PIG-2884: JobControlCompiler mis-logs after reducer estimation (billgraham)
PIG-2876: Bump up Xerces version (jcoveney)
PIG-2866: PigServer fails with macros without a script file (billgraham)
PIG-2860: [piggybank] TestAvroStorageUtils.testGetConcretePathFromGlob fails on some version of hadoop (cheolsoo via jcoveney)
PIG-2861: PlanHelper imports org.python.google.common.collect.Lists instead of org.google.common.collect.Lists (jcoveney)
PIG-2849: Errors in document Getting Started (miyakawataku via billgraham)
PIG-2843: Typo in Documentation (eric59 via billgraham)
PIG-2841: Inconsistent URL in Docs (eric59 via billgraham)
PIG-2740: get rid of "java[77427:1a03] Unable to load realm info from SCDynamicStore" log lines when running pig tests (julien)
PIG-2839: mock.Storage overwrites output with the last relation written when storing UNION (julien)
PIG-2840: Fix SchemaTuple bugs (jcoveney)
PIG-2842: TestNewPlanOperatorPlan fails when new Configuration() picks up a previous minicluster conf file (julien)
PIG-2827: Unwrap exception swallowing in TOP (haitao.yao via jcoveney)
PIG-2825: StoreFunc signature setting in LogicalPlan broken (jcoveney)
PIG-2815: class loader management in PigContext (rangadi via jcoveney)
PIG-2813: Fix test regressions from PIG-2632 (jcoveney)
PIG-2806: Fix merge join test regression from PIG-2632 (jcoveney)
PIG-2809: TestUDFContext broken by PIG-2699 (julienledem via daijy)
PIG-2807: TestParser TestPigStorage TestNewPlanOperatorPlan broken by PIG-2699 (julienledem via daijy)
PIG-2782: Specifying sorting field(s) at nightly.conf (cheolsoo via daijy)
PIG-2790: After Pig-2699 the script schema (LOAD ... USING ... AS {script schema}) is passed after getSchema is called (daijy)
PIG-2777: Docs are broken due to malformed xml after PIG-2673 (dvryaboy)
PIG-2593: Filter by a boolean value does not work (jay23jack via daijy)
PIG-2665: Bundled Jython jar in Pig 0.10.0-RC breaks module import in Python scripts with embedded Pig Latin (daijy)
PIG-2736: Support implicit cast from bytearray to boolean (jay23jack via daijy)
PIG-2508: PIG can unpredictably ignore deprecated Hadoop config options (thw via dvryaboy)
PIG-2691: Duplicate TOKENIZE schema (jay23jack via azaroth)
PIG-2173: piggybank datetime conversion javadocs not properly formatted (hluu via daijy)
PIG-2709: PigAvroRecordReader should specify which file has a problem when throwing IOException (mpercy via daijy)
PIG-2640: Usage message gives wrong information for Pig additional jars (prkommireddi via daijy)
PIG-2652: Skew join and order by don't trigger reducer estimation (dvryaboy)
PIG-2616: JobControlCompiler.getInputSizeFromLoader must handle exceptions from LoadFunc.getStatistics (billgraham)
PIG-2644: Piggybank's HadoopJobHistoryLoader throws NPE when reading broken history file (herberts via daijy)
PIG-2627: Custom partitioner not set when POSplit is involved in Plan (aniket486 via daijy)
PIG-2596: Jython UDF does not handle boolean output (aniket486 via daijy)
PIG-2649: org.apache.pig.parser.ParserValidationException does not expose the cause exception
PIG-2540: [piggybank] AvroStorage can't read schema on amazon s3 in elastic mapreduce (rjurney via jcoveney)
PIG-2618: e2e local fails to build
PIG-2608: Typo in PigStorage documentation for source tagging (prkommireddi via daijy)
PIG-2590: running ant tar and rpm targets on same copy of pig source results in problems (thejas)
PIG-2581: HashFNV inconsistent/non-deterministic due to default platform encoding (prkommireddi via daijy)
PIG-2514: REGEX_EXTRACT not returning correct group with non greedy regex (romainr via daijy)
PIG-2532: Registered classes fail deserialization in frontend (traviscrawford via julien)
PIG-2549: org.apache.pig.piggybank.storage.avro - Broken documentation link for AvroStorage (chrisas via daijy)
PIG-2322: varargs functions do not get passed the arguments in Python embedding (julien)
PIG-2491: Pig docs still mention hadoop-site.xml (daijy)
PIG-2504: Incorrect sample provided for REGEX_EXTRACT (prkommireddi via daijy)
PIG-2502: Make "hcat.bin" configurable in e2e test (daijy)
PIG-2501: Changes needed to contrib/piggybank/java/build.xml in order to build piggybank.jar with Hadoop 0.23
(ekoontz via daijy)
PIG-2499: Pig TestGrunt.testShellCommand occasionally fails (tomwhite via daijy)
PIG-2326: Pig minicluster tests can not be run from eclipse (julienledem via daijy)
PIG-2432: Eclipse .classpath file is out of date (gates)
PIG-2427: getSchemaFromString throws away the name of the tuple that is in a bag (jcoveney via dvryaboy)
PIG-2425: Aggregate Warning does not work as expected on Embedding Pig in Java 0.9.1 (prkommireddi via thejas)
PIG-2384: Generic Invokers should use PigContext to resolve classes (dvryaboy)
PIG-2379: Bug in Schema.getPigSchema(ResourceSchema rSchema) improperly adds two level access (jcoveney via dvryaboy)
PIG-2355: ant clean does not clean e2e test build artifacts (daijy)
PIG-2352: e2e test harness' use of environment variables causes unintended effects between tests (gates)
Release 0.10.1
BUG FIXES
PIG-3107: bin and autocomplete are missing in src release (daijy)
PIG-3106: Missing license header in several java file (daijy)
PIG-3099: Pig unit test fixes for TestGrunt(1), TestStore(2), TestEmptyInputDir(3) (vikram.dixit via daijy)
PIG-2953: "which" utility does not exist on Windows (daijy)
PIG-2960: Increase the timeout for unit test (daijy)
PIG-2942: DevTests, TestLoad has a false failure on Windows (jgordon via daijy)
PIG-2801: grunt "sh" command should invoke the shell implicitly instead of calling exec directly with the command tokens
(jgordon via daijy)
PIG-2798: pig streaming tests assume interpreters are auto-resolved (jgordon via daijy)
PIG-2795: Fix test cases that generate pig scripts with "load " + pathStr to encode "\" in the path (jgordon via daijy)
PIG-2796: Local temporary paths are not always valid HDFS path names (jgordon via daijy)
Release 0.10.0