Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically flatten objects when subobjects:false #97972

Merged
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
08ae2f4
test plus first implementation
piergm Jul 14, 2023
1f03367
improved implementation, plus added more tests
piergm Jul 19, 2023
663dfa9
logic for dotted field names when subobject: false
piergm Jul 20, 2023
fdefcf2
aligned existing test to new logic
piergm Jul 24, 2023
5773464
working dynamic templates
piergm Jul 25, 2023
99bb79e
code lint
piergm Jul 25, 2023
b6bd7be
Merge branch 'elastic:main' into automatically-flatten-objects-when-s…
piergm Jul 25, 2023
8bf797d
added comprehensive tests for the functionality
piergm Jul 26, 2023
dbc8872
added support for array with mixed content
piergm Jul 26, 2023
7cbaa8e
code lint and more tests checks
piergm Jul 26, 2023
50698c1
Update docs/changelog/97972.yaml
piergm Jul 26, 2023
6cd5af2
Merge branch 'elastic:main' into automatically-flatten-objects-when-s…
piergm Jul 27, 2023
21c2be5
re-introduced assertion as per PR comment
piergm Jul 27, 2023
ca624b1
added test that indexes multiple docs with different way to specify t…
piergm Jul 27, 2023
c4f6e3c
re-introduced removed checks for nestedObjects and ObjectMapper if su…
piergm Jul 27, 2023
c8b5cc5
re-introduced removed flag for WithinLeafObject
piergm Jul 28, 2023
c65b593
added test for mapped field that can parse objects nativelly
piergm Jul 28, 2023
2f47686
corrected test that ingest multiple docs with different way to specif…
piergm Jul 28, 2023
f4d725f
Merge branch 'elastic:main' into automatically-flatten-objects-when-s…
piergm Jul 28, 2023
418e67c
added method to identify if a FieldMapper can parse object
piergm Aug 8, 2023
ee633ec
wrong behaviour on removing path and now we return the removed path
piergm Aug 8, 2023
31fc136
added further tests
piergm Aug 8, 2023
f81b356
clean code
piergm Aug 8, 2023
359cefa
added test for corner-case
piergm Aug 8, 2023
73c2d9c
implementation to manage corner-cases
piergm Aug 8, 2023
f350af4
Merge branch 'elastic:main' into automatically-flatten-objects-when-s…
piergm Aug 8, 2023
f34d7f6
added more fields that can parse objects
piergm Aug 9, 2023
2ddfd45
type checks on fields
piergm Aug 9, 2023
063a46c
added comment to clarify code
piergm Aug 9, 2023
11b9324
added comment to clarify code
piergm Aug 9, 2023
d140c24
check lookup dynamic template only if dynamic:true
piergm Aug 9, 2023
eb24547
renamed variable
piergm Aug 9, 2023
a76a5d4
Merge branch 'elastic:main' into automatically-flatten-objects-when-s…
piergm Aug 9, 2023
4bee7f5
small changes due to PR comments
piergm Aug 21, 2023
dba4a9b
renamed method
piergm Aug 21, 2023
f59f2c7
removed method from deprecated class
piergm Aug 21, 2023
b5ebfca
merged main, resolved conflicts
piergm Aug 22, 2023
2ff0de4
added dottedFieldName tests
piergm Aug 22, 2023
b69502e
merged main, resolved conflicts
piergm Aug 22, 2023
0edae22
added JavaDocs for ContentPath
piergm Aug 22, 2023
e571db6
initial implementation after PR comments
piergm Aug 24, 2023
3c623db
all tests passing
piergm Aug 24, 2023
2e483f3
cleaned code
piergm Aug 24, 2023
0560da8
removed comments
piergm Aug 24, 2023
6a5b9e2
reverted changes
piergm Aug 24, 2023
060dd7f
avoid using deprecated methods
piergm Aug 24, 2023
a400ea4
added comments
piergm Aug 24, 2023
6c7a445
simplified ContentPath
piergm Aug 24, 2023
4e62a65
further semplifications
piergm Aug 24, 2023
3ed03b2
JavaDocs + Tests
piergm Aug 24, 2023
765b6f1
Merge branch 'elastic:main' into automatically-flatten-objects-when-s…
piergm Aug 24, 2023
cfab9e2
typos + small change in test
piergm Aug 24, 2023
df992df
renamed class
piergm Aug 24, 2023
a6c8c6b
enhanced test
piergm Aug 24, 2023
c145e7f
small refactor in test
piergm Aug 24, 2023
86736cf
renamed method
piergm Aug 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/changelog/97972.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
pr: 97972
summary: Automatically flatten objects when subobjects:false
area: Mapping
type: enhancement
issues:
- 88934
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ public final class ContentPath {

private String[] path = new String[10];

private int dottedFieldNameIndex = 0;

private String[] dottedFieldName = new String[10];

private boolean withinLeafObject = false;

public ContentPath() {
Expand All @@ -27,20 +31,37 @@ public ContentPath() {
public void add(String name) {
path[index++] = name;
if (index == path.length) { // expand if needed
expand();
expandPath();
}
}

public void addDottedFieldName(String name) {
dottedFieldName[dottedFieldNameIndex++] = name;
if (dottedFieldNameIndex == dottedFieldName.length) {
expandDottedFieldName();
}
}

private void expand() {
private void expandPath() {
String[] newPath = new String[path.length + 10];
System.arraycopy(path, 0, newPath, 0, path.length);
path = newPath;
}

private void expandDottedFieldName() {
String[] newDottedFieldName = new String[dottedFieldName.length + 10];
System.arraycopy(dottedFieldName, 0, newDottedFieldName, 0, dottedFieldName.length);
dottedFieldName = newDottedFieldName;
}

public void remove() {
path[index--] = null;
javanna marked this conversation as resolved.
Show resolved Hide resolved
}

public void removeDottedFieldName() {
dottedFieldName[dottedFieldNameIndex--] = null;
}

public void setWithinLeafObject(boolean withinLeafObject) {
this.withinLeafObject = withinLeafObject;
}
Expand All @@ -58,6 +79,15 @@ public String pathAsText(String name) {
return sb.toString();
}

public String dottedFieldName(String name) {
piergm marked this conversation as resolved.
Show resolved Hide resolved
sb.setLength(0);
for (int i = 0; i < dottedFieldNameIndex; i++) {
sb.append(dottedFieldName[i]).append(DELIMITER);
}
sb.append(name);
return sb.toString();
}

public int length() {
return index;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -306,11 +306,11 @@ private static void innerParseObject(DocumentParserContext context) throws IOExc
parseArray(context, currentFieldName);
break;
case VALUE_NULL:
parseNullValue(context, currentFieldName);
parseNullValue(context, context.path().dottedFieldName(currentFieldName));
break;
default:
if (token.isValue()) {
parseValue(context, currentFieldName);
parseValue(context, context.path().dottedFieldName(currentFieldName));
}
break;
}
Expand Down Expand Up @@ -430,7 +430,15 @@ private static void parseObject(final DocumentParserContext context, String curr
context.path().setWithinLeafObject(false);
context.path().remove();
} else {
parseObjectDynamic(context, currentFieldName);
DynamicTemplate dynamicTemplate = context.findDynamicTemplate(currentFieldName, DynamicTemplate.XContentFieldType.OBJECT);
if (context.parent().subobjects() || dynamicTemplate != null) {
piergm marked this conversation as resolved.
Show resolved Hide resolved
parseObjectDynamic(context, currentFieldName);
} else {
context.parser().nextToken(); // Skipping Object-start
context.path().addDottedFieldName(currentFieldName);
innerParseObject(context);
piergm marked this conversation as resolved.
Show resolved Hide resolved
context.path().removeDottedFieldName();
}
}
}

Expand Down Expand Up @@ -543,12 +551,12 @@ private static void parseNonDynamicArray(DocumentParserContext context, final St
} else if (token == XContentParser.Token.START_ARRAY) {
parseArray(context, lastFieldName);
} else if (token == XContentParser.Token.VALUE_NULL) {
parseNullValue(context, lastFieldName);
parseNullValue(context, context.path().dottedFieldName(lastFieldName));
} else if (token == null) {
throwEOFOnParseArray(arrayFieldName, context);
} else {
assert token.isValue();
parseValue(context, lastFieldName);
parseValue(context, context.path().dottedFieldName(lastFieldName));
}
}
}
Expand Down
Loading