[Feature Request]: <Pagination> to handle very large numbers of items

[s100]

Summary

We would like some sort of mechanism to make it so that when there are, say, 40,000,000 items to page through, at 25 items per page <Pagination> does not attempt to generate a dropdown with 1,600,000 options in it.

Justification

We recently had a customer hit exactly this issue. They had 40,000,000 items and the <Pagination> component was causing their browser to apparently stall or crash while attempting to generate a dropdown list of all the pages. Simply omitting the <Pagination> and showing an apologetic message is one option, but it would be nice to do a little better than that.

Desired UX and success metrics

I don't have a full picture of how I would like this to look, beyond "it shouldn't crash the browser, no matter how many items there are"... capping the dropdown size at 1000 entries would be an acceptable start. This is what we are currently doing, manually.

A better approach would be to have some kind of exponential navigation where we are not provided with access to every page, but a representative sample which is sufficient for us to navigate. Say, pages 1-10, 20, 30, 40, ..., 100, 200, 300, ... 1,599,900, 1,599,910, ..., 1,599,980, 1,599,990-1,600,000. If the current page number is somewhere in the middle, provide exponential options around that point, too.

Required functionality

The existing functionality of the <Pagination> component is acceptable. This I think is mainly an alteration to how it generates its page selections.

Specific timeline issues / requests

No specific timeline... we are working around the problem right now.

Available extra resources

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[tay1orjones]

Hi @s100, this has come up a few times

• Pagination component hung up when totalitems is a big number #6836
• Pagination "totalItems" bigger then 2000000 is blocking whole application #5129
• DataTable: select all action is slow and block ui when the table has thousands of rows even with pagination #6146
• Page selector is not suitable for tables with many pages #4365

The easiest way to prevent this is to cap the amount of data sent to the table via your backend/API. Alternatively you could prevent the totalItems prop from being set to a number that causes the browser to break, and limit the amount of pages put into the select.
I think something like this could work:

// if we had a prop called `maxPages` and it was set to 100 by 
// default, totalItems could be calculated based on the selected pageSize

totalItems={
  totalItems < maxPages * pageSize
    ? totalItems
    : maxPages * pageSize
}

In the future Pagination may use a number input instead of a select to avoid this issue altogether. It's been discussed as a potential part of the new refactor of Pagination, unstable_Pagination:

• Allow to input pages in Pagination component #6884
• Design for pagination number input  #5546

The page select dropdown can alternatively be removed altogether preventing the issue.

[tay1orjones]

I'm going to close this for now as I think my previous comment covers the use case of this pretty well. Let me know if not, we can always continue the discussion and reopen if necessary. Thanks for adding your experience as a data point on the topic - 40,000,000 items is a ton! 😵

[s100]

Just to clarify about our use case: our customer does, in fact, have a database table with 40,000,000+ rows in it. At one row of data per second, it only takes a year and a bit to reach this figure. We're not at liberty to tell our customer to simply have less data, and it's not ideal to just lie to the <Pagination> component about how many items there really are. For example, if we set maxPages = 1000 then the customer will see something like "Items 1 to 25 of 25,000"... and they will want to know what happened to their other 39,975,000+ rows of data.

Also, please note that we are not, of course, actually sending all 40,000,000+ rows of data to the browser for rendering. The way our application works is, it makes a query for one specific page of the data, and the response includes just the 25 or so rows which make up that one page of data, which we then put in a <DataTable>. The response also includes a totalItems property so we can render <Pagination> properly. Changing pages causes a new query to be made. (Note that getting sorting to work in this scenario is quite difficult. We have had to hack the way <DataTable> works quite extensively in order to make it launch new queries on sorting, instead of its default behaviour of assuming that all the data to be sorted is in memory. But that's another story.)

As a workaround, currently we are lying to <Pagination> about what the true value of totalItems is, and then also jury-rigging the itemRangeText, pageRangeText and itemText props to ignore the (incorrect) min/max/total values passed in from <Pagination>, and use the real values instead. It's quite inconvenient, but the result is that <Pagination> displays all of the correct numbers while the dropdown only provides access to the first 1,000 pages.

I look forward to these possible solutions...

[s100]

Well, if anybody has the same problem as us and comes reading this bug, here's what we started doing:

const CappedPagination = ({
  itemRangeText,
  itemText,
  maxPages, // set this to e.g. 1000
  page,
  pageRangeText,
  pageSize,
  pagesUnknown,
  totalItems,
  ...rest
}) => {
  const fakeTotalItems = Math.min(totalItems, maxPages * pageSize)
  const totalPages = Math.ceil(totalItems / pageSize)

  const itemCap = pagesUnknown ? Infinity : totalItems
  const min = Math.min(pageSize * (page - 1) + 1, itemCap)
  const max = Math.min(pageSize * page, itemCap)

  return (
    <Pagination
      totalItems={fakeTotalItems}
      itemRangeText={(fakeMin, fakeMax, fakeTotalItems) => itemRangeText(min, max, totalItems)}
      pageRangeText={(current, fakeTotalPages) => pageRangeText(current, totalPages)}
      itemText={(fakeMin, fakeMax) => itemText(min, max)}
      page={page}
      pageSize={pageSize}
      pagesUnknown={pagesUnknown}
      {...rest}
    />
  )
}

This component shows all the correct output and controls, but caps the page dropdown to just 1000 options.