Change Pre-Fetch Behaviour

README.md

@@ -43,22 +43,27 @@ $conn = \ByJG\AnyDataset\Db\Factory::getDbInstance("mysql://root:[email protected]
 
 - [Getting Started](docs/getting-started.md)
 - [Basic Query and Update](docs/basic-query.md)
+- [Sql Statement Object](docs/sqlstatement.md)
 - [Cache results](docs/cache.md)
 - [Database Transaction](docs/transaction.md)
 - [Load Balance and Connection Pooling](docs/load-balance.md)
 - [Database Helper](docs/helper.md)
+- [Filtering the Query](docs/iteratorfilter.md)
 
 ## Advanced Topics
 
 - [Passing Parameters to PDODriver](docs/parameters.md)
 - [Generic PDO Driver](docs/generic-pdo-driver.md)
 - [Running Tests](docs/tests.md)
+- [Getting an Iterator from an existing PDO Statament](docs/pdostatement.md)
+- [Pre Fetch records](docs/prefetch.md)
 
 ## Database Specifics
 
 - [MySQL](docs/mysql.md)
 - [Oracle](docs/oracle.md)
 - [SQLServer](docs/sqlserver.md)
+- [Literal PDO Connection String](docs/literal-pdo-driver.md)
 
 
 ## Install

docs/basic-query.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 2
+---
+
 # Basics
 
 ## Basic Query

docs/cache.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 4
+---
+
 # Cache results
 
 You can easily cache your results to speed up the results of long queries;

docs/generic-pdo-driver.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 10
+---
+
 # Generic PDO configuration
 
 If you want to use a PDO driver that is not mapped into the `anydataset-db` library you can use the generic PDO driver.

docs/getting-started.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 1
+---
+
 # Getting Started
 
 ## 1. Install the ByJG AnyDatasetDB library

docs/helper.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 7
+---
+
 # Helper - DbFunctions
 
 AnyDataset has a helper `ByJG\AnyDataset\Db\DbFunctionsInterface` that can be return some specific data based on the database connection. 

docs/iteratorfilter.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 8
+---
+
 # Using IteratorFilter
 
 `IteratorFilter` is a class that helps you to create a filter to be used in the Iterator. 

docs/literal-pdo-driver.md

@@ -1,7 +1,14 @@
+---
+sidebar_position: 17
+---
+
 # Literal PDO configuration
 
-If you want to use a PDO driver, and it requires special parameters don't fit well using the URI model you can use the literal object.
-It will allow you to pass the PDO connection string directly.
+If you want to use a PDO driver and this driver is not available in the AnyDatasetDB or and it requires special
+parameters don't fit well
+using the URI model you can use the literal object.
+
+The `PdoLiteral` object uses the PDO connection string instead of the URI model.
 
 Example:
 
@@ -11,6 +18,10 @@ Example:
 $literal = new \ByJG\AnyDataset\Db\PdoLiteral("sqlite::memory:");
 ```
 
-The general rule is use the string as you would use in the PDO constructor.
+Drawbacks:
+
+* You can't use the `Factory::getDbInstance` to get the database instance. You need to use the `PdoLiteral` object
+  directly.
+* The DBHelper won't work with the `PdoLiteral` object.
 
 

docs/load-balance.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 6
+---
+
 # Load balancing
 
 The API have support for connection load balancing, connection pooling and persistent connection.

docs/mysql.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 14
+---
+
 # Driver: MySQL
 
 The connection string can have special attributes to connect using SSL. 

docs/oracle.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 15
+---
+
 # Driver: Oracle
 
 The Oracle Driver don't use the PHP PDO Driver. Instead, uses the OCI library.

docs/parameters.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 9
+---
+
 # Passing Parameters to PDODriver
 
 You can pass parameter directly to the PDODriver by adding to the connection string a query parameter with the value.
@@ -17,14 +21,12 @@ AnyDatasetDB has some special parameters:
 
 | Parameter                      | Value     | Description                                                                                                               |
 |--------------------------------|-----------|---------------------------------------------------------------------------------------------------------------------------|
-| DbPdoDriver::STATEMENT_CACHE   | true      | If this parameter is set with "true", anydataset will cache the last prepared queries.                                    |
 | DbPdoDriver::DONT_PARSE_PARAM  | any value | Is this parameter is set with any value, anydataset won't try to parse the SQL to find the values to bind the parameters. |
 | DbPdoDriver::UNIX_SOCKET       | path      | PDO will use "unix_socket=" instead of "host=".                                                                           |
 e.g.
 
 ```php
 $uri = Uri::getInstanceFromString("sqlite://" . $this->host)
-    ->withQueryKeyValue(DbPdoDriver::STATEMENT_CACHE, "true")
     ->withQueryKeyValue(DbPdoDriver::DONT_PARSE_PARAM, "");
 ```
 

docs/pdostatement.md

@@ -0,0 +1,37 @@
+---
+sidebar_position: 12
+---
+
+# Using a PDO Statement
+
+If you have a PDO Statement created outside the AnyDatasetDB library,
+you can use it to create an iterator.
+
+```php
+<?php
+$pdo = new PDO('sqlite::memory:');
+$stmt = $pdo->prepare('select * from info where id = :id');
+$stmt->execute(['id' => 1]);
+
+$iterator = $this->dbDriver->getIterator($stmt);
+$this->assertEquals(
+    [
+        [ 'id'=> 1, 'iduser' => 1, 'number' => 10.45, 'property' => 'xxx'],
+    ],
+    $iterator->toArray()
+);
+```
+
+Note:
+
+* Although you can use a PDO Statement, it is recommended to use the
+  `SqlStatement` or `DbDriverInterface` to get the Query.
+* Use this feature with legacy code or when you have a specific need to use a PDO Statement.
+
+## Benefits
+
+You can integrate the AnyDatasetDB library with your legacy code and get the benefits of the library
+as for example the standard `GenericIterator` 
+
+
+

docs/prefetch.md

@@ -0,0 +1,64 @@
+---
+sidebar_position: 13
+---
+
+# Pre Fetch records
+
+By Default the records are fetched from the database when you iterate over the records using for example `moveNext()`,
+`toArray()` or `foreach`.
+
+You can pre-fetch a number of records when you get the iterator.
+
+```php
+<?php
+$sql = new SqlStatement("select * from table where field = :param");
+
+// Pre-fetch 100 records
+$iterator = $sql->getIterator($dbDriver, ['param' => 'value'], preFetch: 100);
+```
+
+or
+
+```php
+<?php
+$iterator = $dbDriver->getIterator($dbDriver, ['param' => 'value'], preFetch: 100);
+```
+
+The commands above will fetch 100 records from the database and store in memory.
+When you iterate over the records, it will get the records from memory instead of the database.
+
+## Use cases for pre-fetch:
+
+### Small tables with a few records
+
+If you have a small table with a few records, it is better to fetch all records at once and store in memory
+while you iterate over the records.
+
+### Long processing time
+
+If you have a long processing time between the records, it is better to prefetch the records at once and store in memory
+releasing database resources earlier.
+
+## When not to use pre-fetch
+
+In these cases, it is better to fetch the records from the database without pre-fetching, because you can have memory
+issues:
+
+* if your records are too large (e.g. dozens of columns)
+* If you have a field like a blob or large text
+
+## How it works
+
+When you get the `getIterator` it will fetch the number of records you defined from the database and store in memory.
+When you iterate over the records, it will get the records from memory instead of the database and fetch a new one and
+store in memory.
+
+e.g.:
+
+* You have a table with 60 records and define the preFetch of 50.
+* When you get the iterator, it will fetch 50 records from the database and store in memory.
+* For each record you iterate over, it will get the record from memory and fetch a new one from the database.
+* When you reach the 60th record, the iterator will close the cursor from the database, and allow you fetch the
+  remaining records from memory. 
+
+

docs/sqlserver.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 16
+---
+
 # Driver: Microsoft SQL Server
 
 There are two Drivers to connect to Microsoft SQL Server.

docs/sqlstatement.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 3
+---
+
 # SQL Statement
 
 The SQL Statement is a class to abstract the SQL query from the database.

docs/tests.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 11
+---
+
 # Running Unit tests
 
 ## Unit Tests (no DBConnection)

docs/transaction.md

@@ -1,3 +1,7 @@
+---
+sidebar_position: 5
+---
+
 # Database Transaction
 
 ## Basics

src/DbDriverInterface.php

@@ -23,9 +23,10 @@ public function executeCursor(mixed $statement): void;
      * @param array|null $params
      * @param CacheInterface|null $cache
      * @param int|DateInterval $ttl
+     * @param int $preFetch
      * @return GenericIterator
      */
-    public function getIterator(mixed $sql, ?array $params = null, ?CacheInterface $cache = null, DateInterval|int $ttl = 60): GenericIterator;
+    public function getIterator(mixed $sql, ?array $params = null, ?CacheInterface $cache = null, DateInterval|int $ttl = 60, int $preFetch = 0): GenericIterator;
 
     public function getScalar(mixed $sql, ?array $array = null): mixed;