-
Notifications
You must be signed in to change notification settings - Fork 47
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ORM Discussion #1064
Comments
PSR CaveatsWith respect to PSR-1, coding standard, there is one that really bugs me: "Class constants MUST be declared in all upper case with underscore separators". This creates some pretty ugly code. All upper-case implies shouting in prose, and code should be readable. Its also harder to type, and we should always encourage the use of class contants. I suggest we ignore this one and instead use StudlyCaps instead as we are currently doing. Its obvious something is a contstant when it is preceeded by double colon (::) and does not have parenthesis after it. |
I realized that:
Will not work because sometimes nodes and constants are interchangeable and we detect whether we have a node class or a constant value. Plus we will need namespacing. I think the best alternative to
should be:
with potentially a
so that it can be referred to as
I think referring to this using a method call ->LastName() rather than a __get override is important so that the type safety helps of PHP7 and HHVM can be utilized. It also is slightly more perfomant. |
@spekary You could add one more point to that: Sometimes we need to just increment a field in a row. Think view count - e.g. if we are creating a blog post using QCubed as backend and we want to increase the view count for each access, we should not be needing a full Save() action because that would update the entire column and though it had never happened, it might break the row (especially if the developer makes a mistake). We should allow our ORM to call UPDATE over the row in Such updates are transactional by default and are quite optimised inside the database (at least inside PostgreSQL's source). |
In addition, I also had in mind: Making QCache an interface to other underlying caching technique. Thinking of architecture:We can have all kinds of caching systems under QCache umbrella. So, a developer can declare multiple caches (PHP-Script cache, Memcached, Redis, File-based caching etc.) and use a certain kind of cache according to requirement. So, for example, he/she can use a File-Based cache for images, single object caches using Memcached and Range-Query caches using Redis. So, as @lackovic10 has proposed in another thread - we can use dependency injection in this case. Examples of various cases:Getting PHP object from memorySomething like this could be done (of course, the code can be a lot better/we need to think a little more about this) $objCacher = QCache::GetMemCacher();
$objUser = $objCacher->getUser(1);
if(!$objUser) {
$objUser = User::Load(1);
$objCacher->saveUser(1, $objUser);
} Getting a file from File cache$strFilePath = '/path/to/header/image/for/a/blog/post.png';
$objCacher = QCache::GetFileCacher();
$objFileData = $objCacher->getFileByPath($strFilePath);
if(!$objFileData) {
$objFileData = readfile($strFilePath);
$objCacher->saveFileData($strFilePath, $objFileData);
} ...and so on! |
I was thinking of handling DB procedures as well. To just call a DB method and ask it to verify if a user with given email and password combination exists or not is a lot more secure and convenient on the application level, we can do that. But QCubed (and for that matter any ORM right now) does not support that. If done properly, this can be a very advanced case and be helpful to advanced users. The base work that would go into this can also help us tackle some work with other NoSQL databases (My 5.5th sense says so 😉). |
Regarding incrementing, there is math support now. What we need is better update control. There are lots of situations where you need to so something to just one field in a record that we could do pretty easily. Regarding caching, PSR-6 is a caching interface plan. We should probably move our caching architecture to fit that interface if possible. However, if PSR-6 doesn't fit well with you cache ideas, no need to follow it. |
@spekary Where is the math support? 😕 Also, will go through PSR-6 and will see if we can do something about it. |
QQ::Add, QQ::Sub, etc. These are nodes, so you can select on them. But you can't update on them yet. That would take some improvements to QQuery. |
@spekary great plan! |
There is also a somewhat difficult bug to fix, #753. This one is important to anyone needing to create a datagrid that has data that is part of an array expansion. |
NoSQL SupportA possible strategy for this is to restructure QQuery so that instead of BuildQueryStatement embedded in every ORM object, a query structure is built up, and then passed to each database adapter to unwrap and create the query. All the SQL database adapters could subclass a standard SQL adapter, so that each SQL adapter only needs to implement special cases. A NoSQL adapter can then try to do the query its way. The Intermediate DB definition I mentioned above will help here. Not only will that allow us to know the structure of each entity, but it will allow us to define foreign keys within a NoSQL database. A few things to think about:
|
@spekary I see what you are trying to do. I would advise against it, or rather, I would advise we take it in two different ways. NoSQL is very different from SQL. I mean there are limits on Indexes, they do not understand SQL, there is hardly any generalisation (a little among document-based NoSQL solutions - like Dynamo, Mongo and Couch are similar in ways they accept and present data but their bindings, querying etc. are different). That makes it a bad idea to stuff with QQuery. We should keep NoSQL aloof from RDBMS handling altogether. They just won't fit. About the SQL adapter part - I think that's a Noble idea and I liked it. Had been thinking along those lines (although not as well planned as you, apparently) myself. You can look into Drupal's code for that. They have a nice way of getting that done. |
If we can borrow from other ORMs, great. Re: NoSQL structure, it very much depends on how you use it. Not all NoSQLs are schema-less, and even the ones that are benefit from having some structure in most cases. In the end, QQuery is not supposed to do everything, but solve some common problems. By building such that a NoSQL database could be used to process most QQuery requests (except maybe for aggregate functions on NoSQL dbs that can't do a map reduce), and also building the templates to always use QQuery, then we make it quite possible to use a NoSQL database as a backend for a QCubed code-generated application, including Google cloud datastore. We have to do that anyway if we want to be more SQL database agnostic, but now we can even support most NoSQL databases. Obviously we have to impose some structure to do that with NoSQL, but its still possible and opens up more options for how to build web apps, including using Google App Engine and Amazon Webservices as hosts. |
@spekary Thanks. But I think we should keep the two types separately. For example: If you think of Redis: It's great as a cache. It can be used to store documents and you can run a pattern search over the keys to get more than one object in a query. You can use it a huge datastore to power something as big as reditt. DBs like these with no properly defined usecase (because they are so capable at the simple tasks they handle) can cause big headaches when trying to accomodate functionality into QQuery. If you still insist - let's create a chart of which NoSQL DB can do what and maybe we can come to a conclusion. |
See: http://sailsjs.org/documentation/concepts/models-and-orm/query-language. Their are a couple of node.js orms that try to be database agnostic. I think doing it this way will be really good for the architecture. Yes, NoSQL can be completely random in structure, but many of the NoSQL Databases (MongoDB, FireBase to name a few), encourage you to create a set structure so that you can create indexes on the data.Once you have indexes, you easily get to foreign keys, and now you have something that looks a lot like a relational database. The only difference is the lack of aggregation functions, but that can be handled by the adapters, since each NoSQL has a variety of capabiilities regarding aggregation. It will mean the following:
By doing things this way, you get the following:
|
Look at this https://storm.canonical.com/Tutorial
|
@vmakre this looks similar like doctrine
|
@lackovic10 why you think this?
Nothing new or special here , everything is same as SQL. Xml structure to generate sql for different engines , or nosql databases , why not?. |
can you list the benefits with this approach, why is it better? |
@lackovic10 Good points. I agree. However, there is an approach that can accomodate both. I like Propel's API. But, I don't like being forced to generate from a schema. I found some GO frameworks that will do it in 2 steps - generate a schema from the database, and then generate code from the schema. In addition, you can go backwards, generating the database from the schema. The advantage of the schema is that you can check it in to a source control system. However, if you are creating migration SQL, you would also need to check those in. I find myself having to create migration PHP as well as SQL, so both would need a way of being checked in and related to versions. Since you can go backwards from the schema, you could potentially use this system to migrate to other databases. As I mentioned earlier, there is an ORM called Sails in node.js. They have an interesting approach of being able to use either SQL or NoSQL databases from the same schema. @vaibhav-kaushal pointed out that NoSQL is not really designed for that kind of use, but I have found a number of NoSQL databases that have found the need to build in relational features, and those get us close enough to be useful for QCubed, since QCubed really isn't doing anything particularly difficult in the SQL that couldn't also be done in NoSQL. This doesn't prevent someone from using a NoSQL database in a more traditional sense as a key-value store, but it opens the possibility of being able to migrate from a SQL to a NoSQL database over the life of a project, if that is what the project needs at the time. It could even be done in parts, splitting up the application into objects that are stored in NoSQL and other objects stored in SQL. The 2 step process can let you create a PHP object version of the database for doing code generation. Basically, we do that now, but we generate the objects straight from the database rather than from a schema. A QQuery need currently is to be able to use conditions and clauses to be able to do updates and deletes, not just selects. The problem is that different SQL databases get particularly non-standard when trying to do this. To do this using QQuery, more of the intelligence needs to move to the database adapter so that it can do it in a more database specific way. Another issue is the current bug in how limits interact with array expansions. To really do it correctly requires 2 queries, one to generate the keys of the objects without the array expansion, and another to query for all the data pointed to by these keys and assemble the arrays. Finally, how to do some things SQL does well that NoSQL does not. Some things that come to mind are:
So, here is my proposal: Changes to Codegen
Changes to QQuery Itself
There are a ton of benefits to this, including being able to hand off SQL queries to microservices, use middelware to do logging, use external data stores, perhaps even do it all asynchronously (oooh). We can probably have substantially the same API presented to the application. In other words, the actual model interface does not need to change much. @lackovic10 I did want your thoughts on the generated models. You had some opinions about avoiding static code, and there is copious use of static code there. I can't think of a way around it, but also, not sure its bad. The static code in the models mostly represent different ways to construct objects and arrays of objects out of the database, and static object constructors are used successfully in all kinds of languages and systems. But if you can suggest another way, lets hear it. The other issue is the application object. But maybe more specifically is the database adapters. Do you see a better way than doing what we currently do, calling a static GetDatabase function, that then gets the adapter out of globals? |
wow this seems like a lot of changes in your plan. not even sure if i can weight in, i don't have experience with no sql databases, also quite short of time at the moment to dive deep into this.
this is just a hint of what i have in my mind. when decoupling code into smaller classes with clearly defined dependencies a dependency injection controller is a must. i still need to get back to our discussion about it.
now i inject the
now the queries go into these repository classes, while the other logic from the User class goes to different services, and step by step we expect the user class to have less than a 1000 lines. it had over 10000 if i remember correctly. |
Don't worry too much about the internals for now. We are all busy. Its the external interface I am mostly concerned about. Where do you create a UserRepository? What object owns that? At some point you have to have a static method to create an object. When you start your app out, there are no objects. Using the "new" keyword is just a shortcut for calling a static constructor. For example, this is typical boilerplate code from QCubed:
Lots of statics here. How would you rewrite that using dependency injection? |
the
so again, in a controller or Qform i do
and the container is the only place that worries about managing dependencies. if a new dependency is added to the |
as for the code above
i'm planning to introduce a library that implements PSR7 - message interfaces and use |
i recommend that you try coding this way, you'll see how easy it is to write smaller classes right away |
This is a place to further discuss a standalone ORM.
Goals
Here are some of my goals, some of which have been articulated by others:
Strategies
Standalone
See the #1063 for a proposal on what the namespace and directory structure should be.
The main dependency issues here are the QWatcher and QCache classes. Those should be easily dealt with using configuration settings so that the resulting code optionally does not use these.
Intermediate DB Definition
Currently we allow 2 ways to generate code. One is directly from the database, and one is a kind of hybrid, where the database provides the table structure, but the xml settings file provides the foreign key dependencies. The first method is for inno_db style of SQL, and the second method is for SQL engines that don't have strong foreign key connections.
I propose to separate this. I noticed some Go Lang Orms that use a table definition file, that is then used to drive ORM creation and SQL creation, and it got me thinking that this is part of a good strategy. The thought here is to do the ORM generation in two separate steps:
Advantages include:
Performance
Flexibility
Adding some hooks in to the templates for user-level code generation additions should fix this.
Abstraction
This is mostly providing helps in the stub files and documentation in the examples to guide users think of the ORM as data abstraction only. Also, maybe there are improvements or shortcuts to the QQuery language that we can add to make it easier.
For example, in a typical datagrid binder, you might see:
We need to instead do something or encourage something like:
and then have inside of the People class the QQuery code listed above.
Also, the code inside of People class could look more like:
Also, in the case of someone using a \QQ\Select clause, we should be setting a bit that records that there is valid data in a field so that if someone tries to read data that was not in a QQ::Select, we throw an exception.
These changes will require changes in the UI code generation, hopefully just in the model connector generators. The problem is that the generators currently encourage injection of custom condition and clauses, so the above abstraction may not be completely doable. Filtering and datagrid sorting really require a strong connection between the UI and the data definition. But at least with some of the changes we can abstract away portions of it.
Thoughts?
The text was updated successfully, but these errors were encountered: