Just Another ITS Systems Guy

Some minor updates on the Solaris/AD front. Alex and I have opened tickets with both Microsoft and Sun regarding the problems we’re seeing with VLV support between Solaris and Active Directory. Based on conversations and emails with both companies, we don’t think it’s VLV that’s really broken. Along with VLV, it appears you have to/should implement RFC 2891. What’s RFC 2891 you ask?

“LDAP Control Extension for Server Side Sorting of Search Results”. Also known as SSS.

This function allows the client to tell the server to sort the returned results on some key. The RFC states that the controlling commands that you pass to the server must contain at least one key. But it doesn’t say it must contain only one key. After some pointed searching, Alex was able to find a knowledge base article discussing SSS and how it is implemented on the Active Directory domain controllers. Microsoft has chosen to interpret the RFC as “No, really, we only have to support one, therefore, we’re only going to support one.” That’s what we both inferred from their article.

Sun basically came back and said what we already know: disable VLV on the domain controllers. There’s a hotfix to do this, but it has bad performance impact on Outlook Web Access. Though, now I have to wonder about it because Microsoft made the claim that VLV actually slows things down while on a conference call today. What the heck?! We’re trying to get more clarification on that.

So, we have a good working theory now and the packet captures seem to support it. The Solaris client is throwing two attributes back at AD to sort against: cn and uid. But, AD seems to be kicking back. Hmm. Now that I think about it, I don’t really understand why cn is being used in the sort list. We’re just calling it to fill out gecos information. The only place we reference that in the ldap configuration on the client is to map the GECOS information. I need to do some packet captures and see what happens when I remove that.

In other news, I have done some more work on the Solaris client configuration after somehow managing to break it (again). Before I get too far away from what works, I’ll go ahead and post it here.

# login service (explicit because of pam_dial_auth)
#
login   auth requisite          pam_authtok_get.so.1
login   auth required           pam_dhkeys.so.1
login   auth sufficient         pam_unix_cred.so.1
login   auth sufficient         pam_unix_auth.so.1
login   auth sufficient         pam_krb5.so.1

# Default definitions for Authentication management
# Used when service name is not explicitly mentioned for authentication
#
other   auth requisite          pam_authtok_get.so.1
other   auth required           pam_dhkeys.so.1
other   auth sufficient         pam_krb5.so.1
other   auth required           pam_unix_cred.so.1
other   auth required           pam_unix_auth.so.1 

# Default definition for Account management
# Used when service name is not explicitly mentioned for account management
#
other   account requisite       pam_roles.so.1
other   account required        pam_unix_account.so.1
other   account sufficient      pam_ldap.so.1
other   account required        pam_list.so.1 allow=/etc/user.allow debug

So, what we have here is a PAM configuration that _should_ allow a local user (such as root) or an LDAP user to login to the console. Yes, being able to login as root is important and that was part of my previous breakage that I hadn’t noticed.

The other two pieces, “other account” and “other auth” handle all other services not explicitly named. The auth one is basically the same as the login. The account configuration is special. You’ll see two new PAM modules here: pam_ldap and pam_list. pam_ldap tells PAM to go query against LDAP whenever trying to find account information. I’m not quite sure how it ties into the nsswitch.conf configuration for passwd and group. This was just the recommendation so I went with it.

The pam_list setup is my attempt at mimicking pam_access on Linux. This half works. You should be able to put netgroups into the /etc/user.allow file but because of the VLV/SSS interaction between AD and Solaris, this doesn’t actually query anything useful in LDAP. Herein lies a big problem: I have authentication of my users, but no manageable way to deal with authorization. This is important for some of our customers who need to lock down exactly who can get access to a system once authenticated.

Bummer.

Solving this will require one of three options:

• Microsoft fixes SSS to allow more than one sort key.
• Sun fixes Solaris to allow the client to ignore the VLV support on a server. Not likely unless you’re willing to get your account manager to put an RFE in (or pay for it)
• Run a Microsoft ADAM, mirror your entire AD structure into it, and then turn off VLV support on the ADAM.

So what else is required? We need the ldapclient configuration. Solaris uses a configuration tool to handle this, unlike Linux where you just edit the config file. Solaris builds a config file for you but cautions you to never edit it.

ldapclient manual \
-a credentialLevel=proxy \
-a authenticationMethod=simple \
-a proxyDN=binduser \
-a proxyPassword='binduserpassword' \
-a defaultSearchBase=dc=austin,dc=utexas,dc=edu \
-a domainName=austin.utexas.edu \
-a defaultServerList="AD domaincontroller IP address" \
-a followReferrals=false \
-a attributeMap=group:userpassword=userPassword \
-a attributeMap=group:memberuid=memberUid  \
-a attributeMap=group:gidnumber=gidNumber \
-a attributeMap=passwd:uid=sAMAccountName \
-a attributeMap=passwd:gecos=cn \
-a attributeMap=passwd:gidnumber=gidNumber \
-a attributeMap=passwd:uidnumber=uidNumber \
-a attributeMap=passwd:homedirectory=unixHomeDirectory \
-a attributeMap=passwd:loginshell=loginShell \
-a objectClassMap=group:posixGroup=group \
-a objectClassMap=passwd:posixAccount=user \
-a objectClassMap=shadow:shadowAccount=user \
-a serviceSearchDescriptor='passwd:dc=austin,dc=utexas,dc=edu?sub?uidNumber=*' \
-a serviceSearchDescriptor='shadow:dc=austin,dc=utexas,dc=edu?sub' \
-a serviceSearchDescriptor='user_attr:dc=austin,dc=utexas,dc=edu?sub' \
-a serviceSearchDescriptor='audit_user:dc=austin,dc=utexas,dc=edu?sub' \
-a serviceSearchDescriptor='group:ou=Departments,dc=austin,dc=utexas,dc=edu?sub?gidNumber=*' \
-a serviceSearchDescriptor='netgroup:OU=UNIXTEST,OU=Departments,DC=austin,DC=utexas,DC=edu?one'

There’s also the krb5.conf but that should mostly be the same as the Linux configuration. The Scott Lowe blogs should have the right configuration to start with (though I can post our Solaris one if needed).

One thing I’ve noticed on the Solaris side with this configuration is that if you’re logged in and authenticated via kerberos, you get a wall notice whenever your kerberos ticket is near expiring. I don’t know if this is configurable or not. This is certainly different than the Linux implementation. I think.

BTW, if you’re at UT, I will be providing some more explicit directions for people to play with at the next sync up meeting on this. If you’re not a part of the meeting already, contact me and I’ll talk with you offline.

What a frustrating afternoon. I spent today working on the Solaris LDAP/Kerberos configuration for authenticating against AD. The short story is that I can now successfully login to my Solaris 10 zone using my windows credentials. Just like on the linux box. The long story (with details) will be posted later because Alex and I are still trying to work through an issue with pam_ldap on Solaris. Basically, Windows implements a function called VLV in it’s LDAP config that causes Solaris to choke whenever you attempt to enumerate a large set of directory entries. The “fix” involves disabling it on the domain controllers, but that breaks other things in the Windows environment.

Oh, and my frustration today? Solved with five minutes of work after I had banged on it for three hours. My zone is a test zone. And as a test zone, it never got a DNS entry created for it. Yep. Creating a hostname for it got everything working. What confused me is that I got everything to work for a single login about half way through the day but promptly broke it and had no idea why it worked.

Remember, DNS is an important part of a healthy diet. Who knew?

So I found one downside to using this AD/LDAP configuration. Ok, not really a downside, just a really big caveat. The account used for binding to the LDAP server can get locked out if it authenticates too many times with the wrong password. Discovered this yesterday when I inadvertently changed the password in my configuration while doing some other testing of search options. When things started mysteriously failing soon after, I thought I’d broken my search configuration.

Whoops.

So what did we learn? Be very careful with your bind password. Because of how often we’re binding to the domain controller (and because the bind user is subject to AD policies), it’d be very easy to completely disable your entire authentication environment if you mess this up. Wondering if there’s an alternative way for us to bind to the domain controller, such as using a public/private key instead. New things to investigate.

Well, that was easy enough. Just needed to understand a bit more of the AD OU structure here. (Sanitized a bit for now).

-bash-3.2$ touch foo bar baz quux
-bash-3.2$ ls -l
total 0
-rw-r--r-- 1 hcoyote UNIXTEST-test 0 Jun  3 16:59 bar
-rw-r--r-- 1 hcoyote UNIXTEST-test 0 Jun  3 16:59 baz
-rw-r--r-- 1 hcoyote UNIXTEST-test 0 Jun  3 16:59 foo
-rw-r--r-- 1 hcoyote UNIXTEST-test 0 Jun  3 16:59 quux
-bash-3.2$ id
uid=66000(hcoyote) gid=66000(UNIXTEST-test) groups=66000(UNIXTEST-test)
-bash-3.2$ getent group UNIXTEST-test
UNIXTEST-test:*:66000:hcoyote,member2,member3
-bash-3.2$ getent group
root:x:0:root
bin:x:1:root,bin,daemon
daemon:x:2:root,bin,daemon
sys:x:3:root,bin,adm
adm:x:4:root,adm,daemon
tty:x:5:
disk:x:6:root
lp:x:7:daemon,lp
.
.
.
stapdev:x:101:
stapusr:x:102:
avahi-autoipd:x:103:
UNIXTEST-test:*:66000:hcoyote,effie,csoto

UNIXTEST-test is the group name for gid 66000 in Active Directory. Everything listed before this group comes straight from the local group file because we’re using the appropriate configuration in nsswitch.conf.

This was solved by adding the following to the ldap.conf:

nss_base_group		ou=Departments,?sub?&(objectCategory=group)(gidNumber=*)

Also, you need to modify nsswitch.conf to be:

group: files ldap

One step closer. Next: account authorization via group membership. In other words, only let someone use a resource if they exist in a specific group. Need to figure out if this should be done via netgroup or unix group membership. Off to research!

Ha ha! Further success on the Linux -> Active Directory integration front. I got SSL working for the underlying ldap bind user. What’s this mean? Protection of the directory information over the wire as it travels from the domain controller to the client host where it will be used.

So what’s the necessary setup bits?  There are three options that need to be added to the ldap.conf that I originally came up with.  They are:

ssl yes
tls_cacertfile /etc/ssl/certs/ca_bundle.crt
tls_checkpeer no
uri ldaps://austin.utexas.edu/ (this is a modification from the previous config)

The tls_cacertfile defines the location of the file that contains the certificate authority information used to create the SSL certificate on the Active Directory domain controllers. You need this to verify the authenticity of the dc’s. The file should be .pem formatted and must be converted from the file you retrieve from the internal certificate authority at UT.

Once you’ve downloaded the file, you get it in the DER format which needs to be converted using something like the following.

openssl x509 -in downloadedcert.cer -inform DER -out rootca.pem -outform PEM

Next, you copy the contents of the rootca.pem to the tls_cacertfile file.

Once you’ve configured the ldap.conf with the updated options, you should now be accessing LDAP over SSL. You’ll have to verify this by running something like wireshark and watching the tcp traffic going across the wire. It’ll look something like the stream in the image above.

If you see errors in /var/log/messages that look like the following, then you’ve got something wrong in your configuration still.

Jun  3 14:09:49 fedex getent: nss_ldap: failed to bind to LDAP server ldaps://austin.utexas.edu/: Can't contact LDAP server
Jun  3 14:09:49 fedex getent: nss_ldap: reconnecting to LDAP server (sleeping 4 seconds)...
Jun  3 14:09:53 fedex getent: nss_ldap: failed to bind to LDAP server ldaps://austin.utexas.edu/: Can't contact LDAP server
Jun  3 14:09:53 fedex getent: nss_ldap: reconnecting to LDAP server (sleeping 8 seconds)...
Jun  3 14:10:01 fedex getent: nss_ldap: failed to bind to LDAP server ldaps://austin.utexas.edu/: Can't contact LDAP server
Jun  3 14:10:01 fedex getent: nss_ldap: reconnecting to LDAP server (sleeping 16 seconds)...

Now that we have SSL tackled, time to get group lookups working. I’ll leave that for another posting.

Woot!  With the help of barthag, I got one of our linux boxes configured to provide passwd file map backend via AD/LDAP and authentication via AD/Kerberos.  Most of the problems stem from permissions issues on the AD side and making sure things are open “enough” to let us through to query for information.

On the Linux side (specifically, Red Hat Enterprise Linux 5 Update 2), there are four files you have to touch to make this work: /etc/krb5.conf, /etc/ldap.conf, /etc/nsswitch.conf, and /etc/pam.d/system-auth.

In our configuration, I’m using LDAP to provide the transport for the actual directory information lookups and Kerberos to manage the authenticating of users to the system.  It’s an easy enough configuration for this.  For testing, we’re not encrypting the LDAP connection because we haven’t yet figured out the correcting pathing and formats of the various certificate files necessary for nss_ldap to work correctly (yay underdocumented features!).  Plus, since my test box is close to the AD servers network-wise, I know the connection is secure enough.  Production usage of this config will very likely enforce the use of SSL since we’d be providing these configs for people outside of our local ITS network (but still on-campus).

Also, for this initial test, we’re only working with the passwd map.

So, without further adieu.

/etc/nsswitch.conf

passwd:  files ldap
shadow:  files ldap

/etc/krb5.conf

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = AUSTIN.UTEXAS.EDU
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 forwardable = yes

[realms]
 AUSTIN.UTEXAS.EDU = {
  kdc = austin.utexas.edu:88
  admin_server = austin.utexas.edu:749
  default_domain = austin.utexas.edu
 }

[domain_realm]
 .austin.utexas.edu = AUSTIN.UTEXAS.EDU
 austin.utexas.edu = AUSTIN.UTEXAS.EDU

[appdefaults]
 pam = {
   debug = false
   ticket_lifetime = 36000
   renew_lifetime = 36000
   forwardable = true
   krb4_convert = false
 }

/etc/ldap.conf

###
### ldap config for binding to AD with a read-only bind account over
### cleartext.
###
base dc=austin,dc=utexas,dc=edu
uri ldap://austin.utexas.edu/

binddn thebinduser
bindpw thebindpasswd
scope sub
timelimit 120
bind_timelimit 120
idle_timelimit 3600

nss_initgroups_ignoreusers root,ldap,named,avahi,haldaemon,dbus,radvd,tomcat,radiusd,news,mailman,nscd,gdm
# limit to the Austinites->People OU that have a uid set.
nss_base_passwd		ou=Austinites,ou=People,?sub?&(objectCategory=user)(uid=*)
#nss_base_group		ou=Group,dc=example,dc=com?one
#nss_base_hosts		ou=Hosts,dc=example,dc=com?one
#nss_base_services	ou=Services,dc=example,dc=com?one
#nss_base_networks	ou=Networks,dc=example,dc=com?one
#nss_base_protocols	ou=Protocols,dc=example,dc=com?one
#nss_base_rpc		ou=Rpc,dc=example,dc=com?one
#nss_base_ethers	ou=Ethers,dc=example,dc=com?one
#nss_base_netmasks	ou=Networks,dc=example,dc=com?ne
#nss_base_bootparams	ou=Ethers,dc=example,dc=com?one
#nss_base_aliases	ou=Aliases,dc=example,dc=com?one
#nss_base_netgroup	ou=Netgroup,dc=example,dc=com?one

nss_map_objectclass posixAccount user
nss_map_objectclass shadowAccount user
nss_map_attribute uid sAMAccountName
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_attribute shadowLastChange pwdLastSet
nss_map_objectclass posixGroup group
nss_map_attribute uniqueMember member
pam_login_attribute sAMAccountName
pam_filter objectclass=User
pam_password ad

pam_password_prohibit_message "Sorry, you must change your password using the UTDirect EID interface."

# we don't use referrals at UT.
referrals no

# return more than 10 thousand entries when iterating over the entire
# map.
nss_paged_results yes
page_size 1000

/etc/pam.d/system-auth

%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        sufficient    pam_krb5.so
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        required      pam_deny.so

account     sufficient    pam_unix.so
account     sufficient    pam_krb5.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     required      pam_permit.so

password    requisite     pam_cracklib.so try_first_pass retry=3
password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

Great.  So with this configuration, I can verify that I can login to the Linux box via ssh using my Windows username and do things as myself.  There are some caveats to this:  you must be in the Austinites OU at this point AND you must have the appropriate posixAccount attributes populated.  At a minimum, that’s uidNumber, gidNumber, loginShell, unixHomeDirectory, and uid.  You may have noticed that we’re actually using sAMAccountName for the login name above.  We’re still playing with that.  We might use uid instead because that’s something we can more easily filter on to limit on for different pieces of the account lookups.

So, what’s next?

• need to beat on the system and make sure no one WITHOUT the attributes set can login.
• need to work on the group map and get that working since we’re currently only doing passwd file lookups.
  • this includes understanding how to deal with secondary groups.
• need to work on the netgroup map and come up with a standardized way of handling that so we can (under Linux), configure people in the /etc/security/access.conf for access restrictions.
• need to figure out if there are other access restriction mechanisms we need to pay attention to.
• need to look at all the pam configs on a default box and see if there’s something that isn’t covered by the system-auth template.
• need to figure out just how far into Kerberos we want to go.  What we’re doing now is good for authenticating a single session.  Need to determine if we want to go to single sign-on or not and do all the extra bits associated with using kerberos.
• need to figure out if we should do authentication using LDAP only (honestly, I’d prefer not to, but that depends on what others in the group need).
• need to get this setup on a Solaris 10 system for testing.
• need to address the issue of a person’s loginShell and coming up with some standards for those, in order to deal with departments that have differing shell policies but with overlapping accounts (e.g., researcher that has accounts in two Unix areas but that have differing policies on where or which shells should be used).
• need to address the issue of a person’s unixHomeDirectory possibly differing between two Unix areas.  Same issue as above, but stickier because it’s generally a lot harder to consolidate this down to a consistent path name than it is to consolidate shells down to a consistent path name.

There are problably others, but these are my known unknowns at this point.

Things that were referenced to make all this work:

• http://blog.scottlowe.org/2008/04/11/ad-integration-tip-dealing-with-more-than-1000-users/
• http://blog.scottlowe.org/2007/01/15/linux-ad-integration-version-4/

I was recently in a meeting discussing the integration of Unix account management (passwd and group) with Active Directory via LDAP and Kerberos.  Having done some of this at a previous job, I’d already done some research into getting it working.  I found some useful resources back then on getting this all working right.  I figured I’d share here.

• http://blog.scottlowe.org/2007/01/15/linux-ad-integration-version-4/
• http://blog.scottlowe.org/2007/04/25/solaris-10-ad-integration-version-3/
• http://blog.scottlowe.org/2007/11/27/some-notes-on-solaris-ad-integration/
• http://www.sun.com/bigadmin/features/articles/kerberos_s10.jsp
• http://www-theorie.physik.unizh.ch/~dpotter/howto/kerberos
• http://technet.microsoft.com/en-us/library/bb463172.aspx