Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add utility to iterate over a fetch-joined query #72

Open
Ocramius opened this issue Apr 22, 2020 · 2 comments
Open

Add utility to iterate over a fetch-joined query #72

Ocramius opened this issue Apr 22, 2020 · 2 comments

Comments

@Ocramius
Copy link
Owner

Ocramius commented Apr 22, 2020

As it currently stands, iteration over a DQL query like following will fail due to Query#iterate() not supporting it in ORM (doctrine/orm#5868):

SELECT
    foo,
    bar
FROM
    MyEntity foo
LEFT JOIN
    foo.bars bar

In order to allow that, the ORM (luckily) comes with a relatively smart Paginator component.

I hacked together a prototype of how this could work in following (un-tested) snippet:

    public static function fromPaginator(
        Paginator $paginator,
        EntityManagerInterface $entityManager,
        int $batchSize
    ) : self {
        $currentBatchPosition = (int) $paginator->getQuery()->getFirstResult();
        $maxBatchPosition = min(((int) $paginator->getQuery()->getMaxResults()) + $currentBatchPosition, (int) $paginator->count());

        $nextPage = static function (
            Paginator $paginator,
            int $firstResult,
            int $maxResult
        ) : Paginator {
            $paginatorCopy = clone $paginator;

            $paginatorCopy->getQuery()->setFirstResult($firstResult);
            $paginatorCopy->getQuery()->setMaxResults($maxResult);

            return $paginatorCopy;
        };

        $iterator = (static function () use ($paginator, $currentBatchPosition, $maxBatchPosition, $batchSize, $nextPage) : \Generator {
            $cursorPosition = $currentBatchPosition;

            do {
                $pageWasEmpty = true;

                foreach ($nextPage($paginator, $cursorPosition, $batchSize) as $row) {
                    $pageWasEmpty = false;

                    yield $row;
                }

                if ($pageWasEmpty) {
                    return;
                }
                
                $cursorPosition += $batchSize;
            } while ($maxBatchPosition === 0 || $cursorPosition <= $maxBatchPosition);
        })();

        return self::fromTraversableResult($iterator, $entityManager, $batchSize);
    }

Two problems in the above:

  1. I hate how it is written: it's a mess
  2. unsure if it works for paginating over a query that already has well defined first/max results

We need accurate testing before considering it for inclusion.

@Slamdunk
Copy link

Slamdunk commented Apr 22, 2020

Thanks for the snippet, already pushed it in our 1.400 pods kubernetes cluster 👍

@Ocramius
Copy link
Owner Author

@Slamdunk anxiously waiting for a helm chart for it: no need for tests for that, just an MVP :P

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants