Support for LDAP paged queries

[rpet]

I'm bringing here an enhancement request regarding LDAP paged queries already filed in this two tickets:

• https://sourcesup.renater.fr/tracker/?aid=3525 (opened 2007-11-18)
• https://sourcesup.renater.fr/tracker/?aid=7369 (opened 2010-12-17, duplicate of 3525, provides a patch made by [email protected])

(The bug tracker at renater.fr seems to be restricted to users from French institutions, so I'm not able to contribute)

The rationale behind this request was already documented in those tickets (I'm adding some opinions from my own)

• As an LDAP administrator I'd prefer to use paged searches than raising the search result limit each time we think a query might be approaching that limit.
• In large deployments it could not be feasible to raise the limit to any arbitrary value.
• Paged searches allow for more escalabe deployments
• As an LDAP user I might not have full control over the settings of the LDAP server I have to use
• It's perfectly fair to use LDAP paged seaches. By no means it's a way to "work around legitimate limitations" of any RFC, as it was estated by someone in https://sourcesup.renater.fr/tracker/?aid=3525

[ikedas]

• For example on OpenLDAP, by default, page size limit seems equal to "hard size limit" (size limit of total count of results enforced by server), and not to be enforced by server (cf. Admin manual).
• On Active Directory, page size limit is enforced by MaxPageSize LDAP policy of server, and its value is lower than MaxResultSetSize (size limit of total count of results enforced by server) (cf.
  Doc on technet).

I guess MaxPageSize is the challenge on Active Directory, because its default is slightly low (1000) and it is enforced by server. If several other directory servers also have the same problem, I think introducing paged search seems useful.

[bertaviader]

I was trying to find this link: https://sourcesup.renater.fr/tracker/?aid=7369 but it seems is down. Do you have the code? I need it to find a solution about pagination.
Thanks,
Berta

[lchanouha]

Hello,
Instead of modifiying LDAP server soft limit, or di paged control (my hard limit is higher), I just added sizelimit arg on Sympa/List.pm

    $mesg = $db->do_operation(
        'search',
        base   => "$ldap_suffix",
        filter => "$ldap_filter",
        attrs  => [@attrs],
        scope  => "$source->{'scope'}",
+       sizelimit => 10000
    );

An option (global or per list) would be better.

[rpet]

Instead of modifiying LDAP server soft limit, or di paged control (my hard limit is higher), I just added sizelimit arg on Sympa/List.pm

That is useless when the LDAP server enforces a limit. It also does not scale well.

So supporting paged queries is the way to go IMHO.

[rpet]

Replacing a standard LDAP query with a paged one is pretty straightforward. This is an excerpt of a Perl script I made to query my Active Directory infrastructure:

...

my $ldap = Net::LDAPS->new( $server, onerror => 'die' ) or die "new: $@";

my @controls = ();

# Creates a control for pages of 100 items
my $ctrl_page = Net::LDAP::Control::Paged->new( size => 100 );
push @controls, $ctrl_page;

my @ldap_args = (
	base => $base,
	filter => $filter,
	attrs => @attributes,
	callback => \&processObject,
	control => @controls
);

my $cookie;

while (1) {
	my $mesg = $ldap->search( @ldap_args );

	$mesg->code && die $mesg->error;

	if ($mesg->entries < 1) {
		print STDERR "NO RESULTS\n";
	}

	# Retrieves the page control cookie
	my($resp) = $mesg->control( LDAP_CONTROL_PAGED ) || last;
	$cookie   = $resp->cookie || last;

	# Sets the cookie for the next search
	$ctrl_page->cookie($cookie);
}

[lchanouha]

Instead of modifiying LDAP server soft limit, or di paged control (my hard limit is higher), I just added sizelimit arg on Sympa/List.pm

That is useless when the LDAP server enforces a limit. It also does not scale well.

So supporting paged queries is the way to go IMHO.

Yes you're right this is the best way. sizelimit is a quick workaround at least for openldap for queries between soft and hard limit, if no size.pr is specified, if I understand well openldap documentation.

[ikedas]

If anyone would like to submit a pull request to make this feature request a reality, it would be appreciated.

In my opinion, it is desirable to be able to use the paged operation transparently from the do_operation() method of Sympa::DatabaseDriver::LDAP. To achieve this, directory servers that do not support the paged operation should be considered, such as below:

my $db = Sympa::Database->new('LDAP', ...);

if ($db->__dbh->root_dse->supported_control(
    Net::LDAP::Constant::LDAP_CONTROL_PAGED()
) {
    # Process paged search operations.
} else {
    # Process simple search operation.
}

[ikedas]

Closed as fixed.