Merge branch 'serialization'

appveyor.yml

@@ -1,4 +1,4 @@
-version: 3.9.{build}
+version: 3.10.{build}
 branches:
   only:
   - master

docs/architecture/entity-collections.md

@@ -18,4 +18,4 @@ So taking the above example again, if you were to partition your collections so
 
 ## Where do they live?
 
-So there is an `EntityCollectionManager` which acts as the container for all the entity collections 
+So there is an `EntityDatabase` which acts as the container for all the entity collections 

docs/breaking-changes.md

@@ -1,5 +1,9 @@
 # Breaking Changes
 
+## 3.9.0 -> 3.10.0
+
+- IEntityCollectionManager no longer contains EntityCollections its now within `IEntityDatabase`, which is within there
+
 ## 3.8.0 -> 3.9.0
 
  - `IObservableScheduler` is now known as `IUpdateScheduler` and uses an `ElapsedTime` object not `TimeSpan`

docs/framework/blueprints.md

@@ -6,7 +6,7 @@ how they wish to implement blueprints, as you can add as much configurable prope
 
 Here is an example of a blueprint:
 
-```
+```csharp
 public class PlayerBlueprint : IBlueprint
 {
 	public string Name {get;set;}
@@ -27,7 +27,7 @@ public class PlayerBlueprint : IBlueprint
 Pools are aware of blueprints and you can create an entity with a blueprint to save you the time of having to create the entity 
 then manually applying all the components, which would look like:
 
-```
+```csharp
 var hanSoloEntity = somePool.createEntity(new PlayerBlueprint{ 
 	Name = "Han Solo", 
 	Class = "Smuggler", 
@@ -40,7 +40,7 @@ You have 2 options of applying blueprints to entities, one would be to just new
 `Apply` method passing in the entity, or you could use the available extension methods to apply directly from the entity, 
 this is also chainable so you are able to apply multiple blueprints to the same entity if you wanted, like so:
 
-```
+```csharp
 entity.ApplyBlueprint(new DefaultActorBlueprint())
 	.ApplyBlueprint(new DefaultEquipmentBlueprint())
 	.ApplyBlueprint(new SetupNetworkingBlueprint());

docs/framework/entities.md

@@ -12,11 +12,17 @@ As mentioned entities are created within collections and you can have many entit
 
 ### From `IEntityCollectionManager`
 
-- `myCollectionManager.CreateObservableGroup(myGroup, optionalPoolName)`
+- `myCollectionManager.GetObservableGroup(myGroup, idsForCollectionsToCheck)`
 
 This is probably the most common approach, you get your `IEntityCollectionManager` instance (usually injected in to your class) and you call `CreateObservableGroup`, this will create or return an existing observable group for you which internally contains the `Entities` that match the group for you to query on further. This is often a better approach than accessing entities directly. (read more on this in querying/filtration docs)
 
-- `myCollectionManager.GetEntitiesFor(myGroup, optionalPoolName)`
+- `myCollectionManager.EntityDatabase.*`
+
+The entity collection manager exposes the entity database which can be queried for more info 
+
+### From `IEntityDatabase`
+
+- `entityDatabase.GetEntitiesFor(myGroup, idsForCollectionsToCheck)`
 
 This is not used often but is there for convenience, it allows you to just get back an `IEnumerable<IEntity>` collection which contains all entities which match the group, so it you can query on the matching entities further however you want.
 

docs/framework/groups.md

@@ -18,13 +18,13 @@ There are a few different ways to create a group, here are some of the common wa
 
 There is a `Group` class which implements `IGroup`, this can be instantiated and passed any of the components you want to target, like so:
 
-```c#
+```csharp
 var group = new Group(typeof(SomeComponent));
 ```
 
 There are also some helper methods here so you can add component types if needed via extension methods, like so:
 
-```c#
+```csharp
 var group = new Group()
     .WithComponent<SomeComponent>()
     .WithoutComponent<SomeOtherComponent();
@@ -36,7 +36,7 @@ This is a halfway house between the builder approach and the instantiation appro
 
 So there is also a `GroupBuilder` class which can simplify making complex groups, it is easy to use and allows you to express complex group setups in a fluent manner, like so:
 
-```c#
+```csharp
 var group = new GroupBuilder()
     .WithComponent<SomeComponent>()
     .WithComponent<SomeOtherComponent>()
@@ -49,7 +49,7 @@ So if you are going to be using the same groupings a lot, it would probably make
 
 It is quite simple to make your own group, you just need to implement the 2 getters:
 
-```c#
+```csharp
 public class MyGroup : IGroup
 {
     public IEnumerable<Type> RequiredComponents {get;} =  return new[] { typeof(SomeComponent), typeof(SomeOtherComponent) };

docs/framework/observable-groups.md

@@ -5,7 +5,7 @@ So now you know what entities are and how you can get hold of them, its worth go
 ## Filtration Flow
 
 ```
-IEntityCollectionManager	<-  This contains all collections, which in turn contains ALL entities
+IEntityCollectionManager	<-  This contains the database which contains all collections, which in turn contains ALL entities
      |
      |
 IObservableGroup      		<-  This filters all entities down to only ones which are within the group
@@ -15,7 +15,7 @@ IComputedGroup        		<-  This acts as another layer of filtration on an IObse
                           		i.e Top 5 entities with PlayerComponent sorted by Score
 ```
 
-## IEntityCollectionManager
+## IEntityCollectionManager || IEntityDatabase
 
 The entity collection manager is the root most point where all entity queries should originate from.
 

docs/framework/systems.md

@@ -16,7 +16,7 @@ This is a niche system for when you want to carry out some logic outside the sco
 more fine grained control over how you deal with the entities matched.
 
 Rather than the `SystemExecutor` doing most of the work for you and managing the subscriptions and entity interactions 
-this just provides you the `GroupAccessor` for the entities targetted and its up to you to control how they are 
+this just provides you the `GroupAccessor` for the entities targeted and its up to you to control how they are 
 dealt with.
 
 The `StartSystem` method will be triggered when the system has been added to the executor, and the `StopSystem` 
@@ -30,4 +30,4 @@ So by default (with the default implementation of `ISystemExecutor`) systems wil
 2. Implementations of `IReactToEntitySystem`
 3. Other Systems
 
-However within those groupings it will load the systems in whatever order Zenject (assuming you are using it) provides them, however there is a way to enforce some level of priority by applying the `[Priority(1)]` attribute, this allows you to specify the priority of how systems should be loaded. The ordering will be from lowest to highest so if you have a priority of 1 it will load before a system with a priority of 10.
+However within those groupings it will load the systems in whatever order Zenject/Extenject (assuming you are using it) provides them, however there is a way to enforce some level of priority by applying the `[Priority(1)]` attribute, this allows you to specify the priority of how systems should be loaded. The ordering will be from lowest to highest so if you have a priority of 1 it will load before a system with a priority of 10.

docs/infrastructure/dependency-injection-abstraction.md

@@ -33,7 +33,7 @@ If you want to configure how a binding should work, you can pass into the `Bind`
 ### AsSingleton: `bool`
 
 Setting this to `true` will mean that only one instance of this binding should exist, so if you were to do:
- ```c#
+```csharp
 Bind<IEventSystem, EventSystem>(new BindingConfiguration{AsSingleton = true});
 ```
 Then resolve `IEventSystem` in multiple places you will always get back the same instance, which is extremely handy for infrastructure style objects which should act as singletons. If you provide `false` as the value it will return a new instance for every resolve request.
@@ -46,7 +46,7 @@ This will allow you to give the binding a name for resolving via name.
 
 This allows you to bind to an actual instance of an object rather than a type, which is useful if you need to manually setup something yourself.
 
-```c#
+```csharp
 var someInstance = new Something(foo, bar);
 Bind<ISomething>(new BindingConfiguration{ToInstance = someInstance});
 ```
@@ -55,7 +55,7 @@ Bind<ISomething>(new BindingConfiguration{ToInstance = someInstance});
 
 This allows you to lazy bind something to a method rather than an instance/type, which is useful if you want to setup something in a custom way once all DI configuration has been processed.
 
-```c#
+```csharp
 var bindingConfiguration = new BindingConfiguration({
     ToMethod: container =>
       {
@@ -85,7 +85,7 @@ The configuration object is simple but can be unsightly for larger configuration
 
 There is a builder pattern helper which lets you setup your binding config via a builder rather than an instance of `BindingConfiguration`, this can be used by just creating a lambda within the bind method like so:
 
-```c#
+```csharp
 Bind<ISomething>(config => config
   .AsSingleton()
   .WithName("something-1")
@@ -95,7 +95,7 @@ Bind<ISomething>(config => config
 
 This lets you setup the configuration in a nice way, it also has type safety so you can setup instances and methods using it like so:
 
-```c#
+```csharp
 // With instance
 Bind<ISomething>(config => config
   .ToInstance(new InstanceOfISomething())
@@ -120,7 +120,7 @@ Bind<ISomething>(config => config
 
 There is also another helper which allows you to get an `IObservableGroup` directly from the container, this internally gets the instance of the `IEntityCollectionManager` and requests an observable group of a given type like so:
 
-```c#
+```csharp
 // By group
 var observableGroup = container.ResolveObservableGroup(new MyGroup());
 // By required components

docs/introduction/setup.md

@@ -67,15 +67,18 @@ public abstract class EcsRxApplication
 		// For creating entities, collections, observable groups and managing Ids
 		var entityFactory = new DefaultEntityFactory(new IdPool(), componentRepository);
 		var entityCollectionFactory = new DefaultEntityCollectionFactory(entityFactory);
+		var entityDatabase = new EntityDatabase(entityFactory);
 		var observableGroupFactory = new DefaultObservableObservableGroupFactory();
-		EntityCollectionManager = new EntityCollectionManager(entityCollectionFactory, observableGroupFactory, componentLookup);
+		EntityCollectionManager = new EntityCollectionManager(observableGroupFactory, entityDatabase, componentLookup);
 
 		// All system handlers for the system types you want to support
 		var manualSystemHandler = new ManualSystemHandler(EntityCollectionManager);
+		var basicSystemHandler = new BasicSystemHandler(EntityCollectionManager);
 
 		var conventionalSystems = new List<IConventionalSystemHandler>
 		{
-			manualSystemHandler
+			manualSystemHandler,
+            basicSystemHandler
 		};
 
 		// The main executor which manages how systems are given information

docs/performance/component-type-lookups.md

@@ -8,7 +8,7 @@ So if you are happy to provide a type index/id with your call it will bypass the
 
 So this change means **YOU** have to tell EcsRx ahead of time what indexes/ids to use for your components and provide that data to the rest of your codebase, its easiest to do this by making a class with static int properties like so:
 
-```c#
+```csharp
 public static class ComponentLookupTypes
 {
     public static int NameComponentId = 0;
@@ -19,7 +19,7 @@ public static class ComponentLookupTypes
 
 Then you can just reference these types anywhere which satisfies the telling of the entity the index, and you then just need to make sure when the application is created it uses these explicit lookups rather than auto generating them, generally done by making your own module and loading it like so:
 
-```c#
+```csharp
 public class CustomComponentLookupsModule : IDependencyModule
 {
     public void Setup(IDependencyContainer container)

docs/performance/struct-components.md

@@ -2,7 +2,7 @@
 
 Out the box it assumes you will be using classes for components, i.e:
 
-```c#
+```csharp
 public class MyComponent : IComponent
 {
 	//...

docs/performance/system-affinity.md

@@ -2,7 +2,7 @@
 
 Out the box `ObservableGroups` will just listen to changes across all collections, but you can give them an affinity so they will only listen to changes on certain collections, providing a performance boost as they dont need to listen to changes on entities they will never interact with, however most of the time you are not creating observable groups as they are requested per system. So we need to be able to tell the system what affinity they have so you can have a better suited observable group.
 
-```c#
+```csharp
 // Tell this system that it should only interact with collections with id 1,5,6
 [CollectionAffinity(1,5,6)]
 public class SomeSystemWithAffinity : ISetupSystem

docs/plugins/batched-plugin.md

@@ -10,7 +10,7 @@ Just load the plugin and then extend the required batched system, most of the he
 
 ### Using struct based components
 
-```c#
+```csharp
 public class BatchedMovementSystem : BatchedSystem<PositionComponent, MovementSpeedComponent>
 {
     public BatchedMovementSystem(IComponentDatabase componentDatabase, IComponentTypeLookup componentTypeLookup, IBatchBuilderFactory batchBuilderFactory, IThreadHandler threadHandler) : base(componentDatabase, componentTypeLookup, batchBuilderFactory, threadHandler)
@@ -32,7 +32,7 @@ As you can see we extend the `BatchedSystem` class and provide it the component
 
 The setup is almost identical to the struct based one but instead of using `BatchedSystem` you use `ReferenceBatchedSystem` as shown below:
 
-```c#
+```csharp
 public class BatchedMovementSystem : ReferencedBatchedSystem<PositionComponent, MovementSpeedComponent>
 {
     public BatchedMovementSystem(IComponentDatabase componentDatabase, IComponentTypeLookup componentTypeLookup, IReferenceBatchBuilderFactory batchBuilderFactory, IThreadHandler threadHandler) : base(componentDatabase, componentTypeLookup, batchBuilderFactory, threadHandler)

docs/plugins/computed-plugin.md

@@ -26,7 +26,7 @@ All of these classes are provided as `abstract` classes so you should inherit fr
 
 ### `ComputedGroup`
 
-```c#
+```csharp
 var myGroup = new MyComputedGroup(someObservableGroup); // inherits from ComputedGroup
 foreach(IEntity entity in myGroup) 
 {
@@ -38,7 +38,7 @@ This implements `IComputedGroup` and takes in an `IObservableGroup` instance in
 
 ### `ComputedCollectionFromGroup`
 
-```c#
+```csharp
 var scoresForEntities = new ComputedScores(someObservableGroupWithScores); // inherits from ComputedCollectionFromGroup<int>
 foreach(int someScore in scoresForEntities)
 {
@@ -50,7 +50,7 @@ This provides a way to create a computed collecton based off an observable group
 
 ### `ComputedFromGroup`
 
-```c#
+```csharp
 var partyRating = new ComputedPartyRating(observableGroupOfPartyMemebers); // inherits from ComputedFromGroup<float>
 
 GroupHud.PartyRating.Text = partyRating.Value.ToString();
@@ -60,7 +60,7 @@ This can be useful for taking a group and computing a singular value based upon
 
 ### `ComputedFromData`
 
-```c#
+```csharp
 var firstPlaceRacer = new ComputedFirstPlace(collectionOfRacers); // inherits from ComputedFromData<Racer, IEnumerable<Racer>>
 
 RacerHud.CurrentWinner.Text = firstPlaceRacer.Value.Name;

docs/plugins/persistence-plugin.md

@@ -0,0 +1,73 @@
+# Persistence Plugin
+
+The persistence plugin provides some helpers and conventions around saving/loading entity databases.
+
+As part of this it also adds support for building your own pipelines to send data to various endpoints, which is all built on top of Persistity/LazyData.
+
+- [Persistity](https://github.com/grofit/persistity)
+- [LazyData](https://github.com/grofit/LazyData)
+
+## A bit of background
+
+Historically this functionality all used to be within EcsRx when it was purely a Unity project, as this was an attempt to try and serialize entity/component data from the scene. However Unity is a complex beast and due to various other reasons the serialization stuff got sidelined.
+
+Anyway fast-forward a bit and now EcsRx is cross platform and Unity is just one of the supported frameworks, so now seemed a good time to re-introduce the serialization aspect of it all.
+
+Given the actual functionality developed was pretty self contained it made sense to split them out into their own libraries that could be used outside of EcsRx, which is why they are their own separate repos, but just wanted to let you know that these libraries were specifically developed originally as part of EcsRx to assist with data push/pulling to various places.
+
+## `EcsRxPersistedApplication`
+
+As part of this there is a helper application class, which extends the default `EcsRxApplication` and by default will look for an existing entity database file and load it when the app starts.
+
+If you do not want it to load automatically and want to handle the load yourself you can override `LoadOnStart` or `SaveOnStop` if you dont want to override the entity database. Also as part of this there is some helper methods for saving and loading the entity database.
+
+There is an example which shows how to use this, as well as how to override the default binary file and use a JSON file instead.
+
+## Pipelines In General
+
+For more info on pipelines look at [Persistity](https://github.com/grofit/persistity), but the gist of it is that they are basic ETL (Extract Transform Load) processes, where you specify how the pipeline should operate. Rather than simple serialization where you just take a model and turn it into a format, a pipeline is a level above that, so you express the entire process/pipeline for taking a raw object and converting it into a given format. Out the box you have support for Binary, Xml, Json, Bson, Yaml.
+
+### Example pipeline 
+For example if you wanted to have a pipeline to make an encrypted save game file it may look like:
+
+```csharp
+// Manually create an encryption/decryption processor
+var encryptor = new AesEncryptor("some-pass-phrase");
+var encryptionProcessor = new EncryptDataProcessor(encryptor);
+var decryptionProcessor = new DecryptDataProcessor(encryptor);
+
+// Tell it where to store our stuff
+var fileEndpoint = new FileEndpoint("savegame.sav");
+
+// Create the pipeline to save the data
+container.BuildPipeline("SaveGame", x => x
+        .StartWithInput()
+        .SerializeWith<IBinarySerializer>()
+        .ProcessWith(encryptionProcessor)
+        .ThenSendTo(fileEndpoint));
+
+// Create the pipeline to load the data
+container.BuildPipeline("LoadGame", x => x
+        .StartFrom(fileEndpoint)
+        .ProcessWith(decryptionProcessor)
+        .DeserializeWith<IBinaryDeerializer>());
+
+// Then to use it you would do
+var mySaveGameData = //...
+var savePipeline = container.ResolveSendPipeline("SaveGame");
+savePipeline.Execute(mySaveGameData); // Now encrypted and saved in savegame.sav
+
+// Then to load it
+var loadPipeline = container.ResolveReceivePipeline("LoadGame");
+var mySaveGameData = loadPipeline.Execute();
+```
+
+As you can see you can setup complex pipelines which can process data/transform it and send it to files, http endpoints, databases or even store it in memory or raise an event etc.
+
+The above example uses some of the helper functionality which act as extensions on the DI container, but you can also inject `EcsRxPipelineBuilder` manually and use that directly to make a pipeline without injecting it in.
+
+## Some things to know
+
+When using pipelines you need to have objects which have a parameterless constructor and have everything get/settable. If you are using 3rd party objects which have complex constructors you are advised to make a proxy type which you transform to and from in your pipeline. For an example of this approach look at how we have `EntityDatabaseData` which acts as a proxy type for `EntityDatabase` which has a complex constructor.
+
+You may want to have 2 different types of the same pipeline, for example in development you may want to have an item database as a json file, but when you go to production you want it as binary. The only difference between the 2 pipelines would be the transport format, so you can re-use a lot of the same steps but just change the send/receive format, you can even use LazyData directly and just convert from Json -> Binary (As shown with SuperLazy helpers in the LazyData repo).