EL9/CentOS Stream 9 lost offline smart card authentication #7532

opoplawski · 2024-08-12T14:12:11Z

We are starting to run into some issues with offline smart card authentication with EL9/CS9 systems. Currently I have a CS9 laptop that when I brought it home I could no longer log in - I get a "Please (re)insert (different) smartcard" message.

sssd-2.9.5-4.el9.x86_64

Config:

[domain/nwra.com]
cache_credentials = True
krb5_auth_timeout = 30
krb5_store_password_if_offline = True
krb5_realm = NWRA.COM
ipa_domain = nwra.com
id_provider = ipa
auth_provider = ipa
access_provider = ipa
sudo_provider = ipa
autofs_provider = ipa
subdomains_provider = ipa
session_provider = ipa
hostid_provider = ipa
chpass_provider = ipa
ipa_server =SRVER1, SERVER2, _srv_
ldap_tls_cacert = /etc/ipa/ca.crt
dns_discovery_domain = nwra.com
ipa_automount_location = boulder
timeout = 20
debug_level = 8
debug_backtrace_enabled = False

[sssd]
services = nss, pam, ssh, sudo, autofs
domains = nwra.com
full_name_format = %1$s
default_domain_suffix = ad.nwra.com
debug_level = 8

[pam]
debug_level = 8
pam_cert_auth = True
# kscreensaver for EL7, kde for Fedora
pam_p11_allowed_services = +kscreensaver, +kde, +xfce4-screensaver
# To display PIN is blocked message
pam_verbosity = 2

sssd_pam.log:
sssd_pam.log

The text was updated successfully, but these errors were encountered:

opoplawski · 2024-08-12T14:13:10Z

To my untrained eye this seems like it might be relevant:

(2024-08-12  7:58:31): [pam] [cache_req_search_send] (0x0400): [CID#8] CR #35: Object found, but needs to be refreshed.

alexey-tikhonov · 2024-08-12T15:02:22Z

when I brought it home

Does it mean you can't auth online?

From the attached log:

Local auth policy allowed: smartcard [False], passkey [False]

Check if man sssd.conf::local_auth_policy helps.

opoplawski · 2024-08-12T22:46:44Z

Authentication will work when the system is online.

I don't think the local auth policy applies here - all of our users are in AD via an IPA/AD trust.

alexey-tikhonov · 2024-08-13T07:08:57Z

Authentication will work when the system is online.

Sorry, I meant to say: "when I brought it home" == auth happens offline?

I don't think the local auth policy applies here - all of our users are in AD via an IPA/AD trust.

IIRC, if auth happens offline (so can't use PKINIT at your KDC) then it falls back to local auth.

opoplawski · 2024-08-13T14:27:30Z

Right, I can't auth offline. I'm not entirely sure what you mean by "local" auth - it certainly authenticates against cached credentials. All I know is that this has been working for years, and now I'm starting to see this offline failure at times on some of our EL9 system. This is a show-stopper for us and I need to figure out what is going on so we can fix it.

p11child.log
sssd_nwra.com.log

alexey-tikhonov · 2024-08-13T14:40:28Z

Did you try to allow smart card auth in local_auth_policy?

opoplawski · 2024-08-15T13:55:15Z

I added to domain/nwra.com:

local_auth_policy = enable:smartcard

But no difference, I still see:

(2024-08-15  7:03:36): [pam] [pam_reply] (0x0400): [CID#1] Local auth policy allowed: smartcard [False], passkey [False]

On my other laptop that is working I do see:

(2024-08-15  7:10:52): [pam] [pam_reply] (0x0400): [CID#7] Local auth policy allowed: smartcard [True], passkey [False]

So that does seem like the likely culprit. But why is it getting set to False?

opoplawski · 2024-08-15T13:58:08Z

I also tried adding it to domain/nwra.com/ad.nwra.com but got:

[sssd] [sss_ini_call_validators] (0x0020): [rule/allowed_subdomain_options]: Attribute 'local_auth_policy' is not allowed in section 'domain/nwra.com/ad.nwra.com'. Check for typos.

opoplawski · 2024-08-15T14:44:58Z

On working system:

(2024-08-15  8:39:54): [pam] [pam_get_auth_types] (0x4000): [CID#2] Authentication types for user [[email protected]] and service [gdm-smartcard]: password
(2024-08-15  8:39:54): [pam] [pam_reply] (0x0400): [CID#2] Local auth policy allowed: smartcard [True], passkey [False]

on failing system:

(2024-08-15  8:33:07): [pam] [pam_get_auth_types] (0x4000): [CID#1] Authentication types for user [[email protected]] and service [gdm-smartcard]: password
(2024-08-15  8:33:07): [pam] [pam_reply] (0x0400): [CID#1] Local auth policy allowed: smartcard [False], passkey [False]

Not sure why we get the Auth types message if I'm setting it explicitly. I suspect this isn't working for sub-domains.

opoplawski · 2024-08-15T14:54:58Z

Okay, here's the difference:

# ldbsearch -H /var/lib/sss/db/cache_nwra.com.ldb  [email protected] | grep -Fi auth
asq: Unable to register control with rootdse!
localPasskeyAuth: FALSE
lastOnlineAuth: 1722860790
lastOnlineAuthWithCurrentToken: 1722860790
localSmartcardAuth: FALSE

on the working system localSmartcardAuth is TRUE. Why would that be different?

opoplawski · 2024-08-15T15:03:28Z

I changed that in the database and was able to log in. But now need to figure out why it is getting changed.

opoplawski · 2024-08-15T19:31:36Z

Okay, I think I found the trigger. If I authenticate with sudo without a smartcard present, the fact that smartcard authentication was not possible:

(2024-08-15 13:26:16): [pam] [pam_get_auth_types] (0x4000): [CID#9] Authentication types for user [[email protected]] and service [sudo]: password
(2024-08-15 13:26:16): [pam] [pam_reply] (0x0400): [CID#9] Local auth policy allowed: smartcard [False], passkey [False]

gets recorded in the cache database:

# ldbsearch -H /var/lib/sss/db/cache_nwra.com.ldb  [email protected] localSmartcardAUth
asq: Unable to register control with rootdse!
# record 1
dn: [email protected],cn=users,cn=ad.nwra.com,cn=sysdb
localSmartcardAuth: FALSE

opoplawski · 2024-08-15T20:14:04Z

I also see this with ssd-2.9.4-6.el9_4.1.alma.1

opoplawski · 2024-08-15T20:27:11Z

I do NOT see this behavior with sssd-2.9.4-4.el8_10.x86_64

alexey-tikhonov · 2024-08-19T09:02:21Z

Sorry for delays with responses (PTO season) and thanks for the investigation. We will need some time to review this.

sumit-bose · 2024-08-28T15:23:45Z

Hi,

thank you for your report. It is as you have described, if there is not Smartcard inserted and a different authentication method is used the localSmartcardAuth attribute is set to FALSE. The reason is that even if the KDC indicates that Smartcard based authentication (pkinit) is possible the pkinit plugin calls out callback only if a Smartcard or similar is present.

So we either have to find a way to see if the KDC offers pkinit or we should not overwrite localSmartcardAuth unconditionally.

bye,
Sumit

alexey-tikhonov · 2024-08-28T15:52:39Z

https://issues.redhat.com/browse/SSSD-7934

opoplawski · 2024-09-03T19:35:59Z

Any chance I could get access to the redhat issue? Login ID is opoplawski. Thanks.

alexey-tikhonov · 2024-09-04T08:06:12Z

Any chance I could get access to the redhat issue?

This is an internal tracker, sorry that private URL gets published in public comments.

Once RHEL project product ticket (public) will be created, link will be posted here as well.

The krb5 backend will only returns that Smartcard authentication is available if a Smartcard is present. That means if the user authenticates with a different method and a Smartcard is not present at this time 'sc_allow' will be 'false' and might overwrite a 'true' value written during a previous authentication attempt where a Smartcard was present. To avoid this we only write 'true' values. Since the default if SYSDB_LOCAL_SMARTCARD_AUTH is missing is 'false' local Smartcard authentication (offline) will still only be enabled if online Smartcard authentication was detected. Resolves: SSSD#7532

sumit-bose · 2024-09-18T13:53:31Z

Hi @opoplawski,

I wonder if you can check if the fix from #7602 works for you. You can find a couple of build with the fix at https://copr.fedorainfracloud.org/coprs/g/sssd/pr7602/.

Thanks.

bye,
Sumit

The krb5 backend will only returns that Smartcard authentication is available if a Smartcard is present. That means if the user authenticates with a different method and a Smartcard is not present at this time 'sc_allow' will be 'false' and might overwrite a 'true' value written during a previous authentication attempt where a Smartcard was present. To avoid this we only write 'true' values. Since the default if SYSDB_LOCAL_SMARTCARD_AUTH is missing is 'false' local Smartcard authentication (offline) will still only be enabled if online Smartcard authentication was detected. Resolves: #7532 Reviewed-by: Iker Pedrosa <[email protected]> Reviewed-by: Justin Stephenson <[email protected]> (cherry picked from commit 67ba42c)

alexey-tikhonov · 2024-09-20T09:16:16Z

Pushed PR: #7602

master
- 67ba42c - pam: only set SYSDB_LOCAL_SMARTCARD_AUTH to 'true' but never to 'false'.
sssd-2-9
- b4c4968 - pam: only set SYSDB_LOCAL_SMARTCARD_AUTH to 'true' but never to 'false'.

alexey-tikhonov · 2024-09-20T09:20:33Z

Hi @opoplawski,

I wonder if you can check if the fix from #7602 works for you. You can find a couple of build with the fix at https://copr.fedorainfracloud.org/coprs/g/sssd/pr7602/.

Sorry I missed this request.
Now PR was merged and corresponding copr builds removed.

@opoplawski, you still can use a nightly build from https://copr.fedorainfracloud.org/coprs/g/sssd/nightly/ (once it gets rebuild with 67ba42c / b4c4968 )

opoplawski · 2024-09-24T02:01:02Z

Looks like none of the nightly builds are succeeding.

alexey-tikhonov · 2024-09-24T06:36:11Z

Looks like none of the nightly builds are succeeding.

Huh...

https://download.copr.fedorainfracloud.org/results/@sssd/nightly/centos-stream-9-x86_64/08042357-sssd/builder-live.log.gz

Finish: build setup for sssd-2.10.0~beta2-0.240920.091653.git67ba42c48.el9.src.rpm
Start: rpmbuild sssd-2.10.0~beta2-0.240920.091653.git67ba42c48.el9.src.rpm
Building target platforms: x86_64
Building for target x86_64
setting SOURCE_DATE_EPOCH=1611187200
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.66rcEe
+ umask 022
+ cd /builddir/build/BUILD
+ cd /builddir/build/BUILD
+ rm -rf sssd-2.10.0-beta2
+ /usr/bin/gzip -dc /builddir/build/SOURCES/sssd-2.10.0-beta2.tar.gz
+ /usr/bin/tar -xof -
/usr/bin/tar: This does not look like a tar archive
/usr/bin/tar: Exiting with failure status due to previous errors

https://download.copr.fedorainfracloud.org/results/@sssd/nightly/fedora-rawhide-x86_64/08042357-sssd/builder-live.log.gz

Finish: build setup for sssd-2.10.0~beta2-0.240920.091653.git67ba42c48.fc42.src.rpm
Start: rpmbuild sssd-2.10.0~beta2-0.240920.091653.git67ba42c48.fc42.src.rpm
Building target platforms: x86_64
Building for target x86_64
setting SOURCE_DATE_EPOCH=1611187200
Executing(%mkbuilddir): /bin/sh -e /var/tmp/rpm-tmp.X3ZJca
+ umask 022
+ cd /builddir/build/BUILD/sssd-2.10.0_beta2-build
+ test -d /builddir/build/BUILD/sssd-2.10.0_beta2-build
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w /builddir/build/BUILD/sssd-2.10.0_beta2-build
+ /usr/bin/rm -rf /builddir/build/BUILD/sssd-2.10.0_beta2-build
+ /usr/bin/mkdir -p /builddir/build/BUILD/sssd-2.10.0_beta2-build
+ /usr/bin/mkdir -p /builddir/build/BUILD/sssd-2.10.0_beta2-build/SPECPARTS
+ RPM_EC=0
++ jobs -p
+ exit 0
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.7HDSIv
+ umask 022
+ cd /builddir/build/BUILD/sssd-2.10.0_beta2-build
+ cd /builddir/build/BUILD/sssd-2.10.0_beta2-build
+ rm -rf sssd-2.10.0-beta2
+ /usr/lib/rpm/rpmuncompress -x /builddir/build/SOURCES/sssd-2.10.0-beta2.tar.gz
/usr/bin/tar: This does not look like a tar archive

etc

@pbrezina, somehow nightly copr builds are broken (while PR copr builds are working)...

pbrezina · 2024-09-24T10:02:59Z

The tarball is empty, but I don't know why.

Pull requests use different code to generate the srpm. Nightly build use make srpm

pbrezina · 2024-09-24T10:07:02Z

COPR nigthly use https://github.com/SSSD/sssd/blob/master/.copr/Makefile but if I run it locally it works nicely.

pbrezina · 2024-09-24T10:11:55Z

There's

./contrib/fedora/make_srpm.sh --prerelease --output /mnt/safe-resultdir-c3dy38x1
remote: fatal: detected dubious ownership in repository at '/mnt/workdir-a7gx95j6/sssd/.git'        
remote: To add an exception for this directory, call:        
remote: 
remote: 	git config --global --add safe.directory /mnt/workdir-a7gx95j6/sssd/.git        
remote: git upload-archive: archiver died with error
fatal: sent error to the client: git upload-archive: archiver died with error

In the logs, I'm contacting copr maintainers.

pbrezina · 2024-09-24T11:06:00Z

fedora-copr/copr#3421

wiad · 2024-10-16T10:54:58Z

We are also seeing this, when will this patch be released in RHEL9?

alexey-tikhonov · 2024-10-16T11:03:06Z

We are also seeing this, when will this patch be released in RHEL9?

https://issues.redhat.com/browse/RHEL-59876

As batch#0 update with RHEL 9.5

alexey-tikhonov added Bugzilla SSSD-Jira Already cloned to SSSD Jira for tracking task breaking down purposes Future work and removed Bugzilla labels Aug 28, 2024

sumit-bose mentioned this issue Sep 18, 2024

pam: only set SYSDB_LOCAL_SMARTCARD_AUTH to 'true' but never to 'false'. #7602

Closed

alexey-tikhonov closed this as completed in 67ba42c Sep 20, 2024

alexey-tikhonov added the Closed: Fixed Issue was closed as fixed. label Sep 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EL9/CentOS Stream 9 lost offline smart card authentication #7532

EL9/CentOS Stream 9 lost offline smart card authentication #7532

opoplawski commented Aug 12, 2024 •

edited

Loading

opoplawski commented Aug 12, 2024

alexey-tikhonov commented Aug 12, 2024 •

edited

Loading

opoplawski commented Aug 12, 2024

alexey-tikhonov commented Aug 13, 2024

opoplawski commented Aug 13, 2024

alexey-tikhonov commented Aug 13, 2024

opoplawski commented Aug 15, 2024 •

edited

Loading

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

alexey-tikhonov commented Aug 19, 2024

sumit-bose commented Aug 28, 2024

alexey-tikhonov commented Aug 28, 2024

opoplawski commented Sep 3, 2024

alexey-tikhonov commented Sep 4, 2024 •

edited

Loading

sumit-bose commented Sep 18, 2024

alexey-tikhonov commented Sep 20, 2024

alexey-tikhonov commented Sep 20, 2024

opoplawski commented Sep 24, 2024

alexey-tikhonov commented Sep 24, 2024

pbrezina commented Sep 24, 2024 •

edited

Loading

pbrezina commented Sep 24, 2024

pbrezina commented Sep 24, 2024

pbrezina commented Sep 24, 2024

wiad commented Oct 16, 2024

alexey-tikhonov commented Oct 16, 2024

EL9/CentOS Stream 9 lost offline smart card authentication #7532

EL9/CentOS Stream 9 lost offline smart card authentication #7532

Comments

opoplawski commented Aug 12, 2024 • edited Loading

opoplawski commented Aug 12, 2024

alexey-tikhonov commented Aug 12, 2024 • edited Loading

opoplawski commented Aug 12, 2024

alexey-tikhonov commented Aug 13, 2024

opoplawski commented Aug 13, 2024

alexey-tikhonov commented Aug 13, 2024

opoplawski commented Aug 15, 2024 • edited Loading

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

opoplawski commented Aug 15, 2024

alexey-tikhonov commented Aug 19, 2024

sumit-bose commented Aug 28, 2024

alexey-tikhonov commented Aug 28, 2024

opoplawski commented Sep 3, 2024

alexey-tikhonov commented Sep 4, 2024 • edited Loading

sumit-bose commented Sep 18, 2024

alexey-tikhonov commented Sep 20, 2024

alexey-tikhonov commented Sep 20, 2024

opoplawski commented Sep 24, 2024

alexey-tikhonov commented Sep 24, 2024

pbrezina commented Sep 24, 2024 • edited Loading

pbrezina commented Sep 24, 2024

pbrezina commented Sep 24, 2024

pbrezina commented Sep 24, 2024

wiad commented Oct 16, 2024

alexey-tikhonov commented Oct 16, 2024

opoplawski commented Aug 12, 2024 •

edited

Loading

alexey-tikhonov commented Aug 12, 2024 •

edited

Loading

opoplawski commented Aug 15, 2024 •

edited

Loading

alexey-tikhonov commented Sep 4, 2024 •

edited

Loading

pbrezina commented Sep 24, 2024 •

edited

Loading