Refactor FATE

[milleruntime]

I was looking at the warnings we suppress in ZooStore and they all seem to stem from the serialization of the generics. I was wondering if it would be better to refactor out the generic types so we don't have to arbitrarily serialize objects and blindly cast them. This would be a bit of work but it might not be too bad. I think if we create an interface like FateEnvironment to pass to the operations we wouldn't have to pass the Manager around.

[EdColeman]

Maybe this should consider #1249

[milleruntime]

If this was for 3.0, we could probably drop the old Bulk Import. That would help clean up a lot of code.

[milleruntime]

One idea I had was to create an Environment interface to pass between FATE and the Manager. The tricky part is that some of the FATE code is in core (the interface, framework and some implementation) while most of it is in the same package as the Manger. I thought we could do this:

public interface RepoEnv {
  public ServerContext getContext();
}

But that won't work because ServerContext is in server and Fate is in core.

[ctubbsii]

On serialization, Java 17 has the new record keyword. It should be safer to serialize records than general Java objects, as we've done in the past. I have not yet used records, and we aren't using Java 17 yet... but could for 3.0 development.

[keith-turner]

It should be safer to serialize records than general Java objects

Looking around for what we could do now to make things safer, found the following. Maybe we could use the new ObjectInputFilter added in Java 9 to explicitly limit what classes FATE can deserialize.

https://www.oracle.com/java/technologies/javase/seccodeguide.html#8
https://docs.oracle.com/javase/9/docs/api/java/io/ObjectInputFilter.html

[milleruntime]

Another improvement that could be included in this change:
Convert FateTxId to a stronger type to prevent coding errors like forgetting to call the method to display the value in HEX.
The code would probably be 10x better if we had a FateTxId type that we used instead of long. Its toString method could consistently format for logging maybe. That could possible encapsulate or extend UUID, but I think we would benefit form a more specific type than UUID or long.

[milleruntime]

For serialization of FATE repos, we could use JSON. There is already this in the FateCommand:

accumulo/shell/src/main/java/org/apache/accumulo/shell/commands/fate/FateCommand.java

Lines 85 to 90 in 0fbf9cd

	private static class ByteArraySerializer implements JsonSerializer<byte[]> {
	@Override
	public JsonElement serialize(byte[] link, Type type, JsonSerializationContext context) {
	return context.serialize(new ByteArrayContainer(link));
	}
	}

[milleruntime]

@ctubbsii @keith-turner I have been experimenting with JSON serialization in FATE. There are already some classes to serialize the data for displaying with the Fate command. What do you guys think of trying to change the serialization to JSON in 2.1?

[ctubbsii]

@ctubbsii @keith-turner I have been experimenting with JSON serialization in FATE. There are already some classes to serialize the data for displaying with the Fate command. What do you guys think of trying to change the serialization to JSON in 2.1?

I think that would probably be too big and too disruptive for 2.1 as we're trying to wrap up major features for 2.1. But, I'm not opposed to the idea, generally. However, I still think it's worth looking into newer Java serialization for record objects in 3.0.

[milleruntime]

@ctubbsii @keith-turner I have been experimenting with JSON serialization in FATE. There are already some classes to serialize the data for displaying with the Fate command. What do you guys think of trying to change the serialization to JSON in 2.1?

I think that would probably be too big and too disruptive for 2.1

After some experimentation, that was my feeling as well. The generics combined with the complexity of the different interfaces really make it difficult to make any small changes.

[dlmarion]

@keith-turner @cshannon - this seems to be about Fate serialization with generics. Is this still an issue with the recent Fate changes?

[cshannon]

@keith-turner @cshannon - this seems to be about Fate serialization with generics. Is this still an issue with the recent Fate changes?

The generics haven't changed, but I don't really think it's a big deal and I personally don't see a need to refactor it and not sure it would actually help anything.