Linux Permissions Headache
Yikes, what an evening….. it started innocently enough in the afternoon. I have an old Mandrake 10.0 server that I was upgrading clamav on (recent security update). While I was at it, I was reviewing the anti-spam setup to see if I could get any better success with filtering junk mail. spamassassin has had an update since I updated this one last and also it seemed that dcc was installed, but not in active use *(no indications that it’s being used at all.) So, I set about trying to fix that and install the latest spamassassin. Somehere along the way something BAD happened. In retrospect, I’m not entirely sure how, but at one point I was root having just installed the rpms for spamassassin and then exited to my user account. Promptly on switching back to my user account I got a “permission denied” error. Eh? Ok, well let’s su again and see what’s up…. “permission denied” uh oh…. ls “permission denied” most everything actually….. permission denied.
What was worse is that I couldn’t initiate a new login session as a user either. Unable to start session was the last error, but couldn’t cd to /home/username was the first message. I could only log in as root and got a permission denied error for su to any other user. Of course, I spent some time googling and found many examples of similar problems. Unfortunately, there seemed scant solid answers as to what might have happened. (Or good examples of “try this” with successful results.) This groups thread seems to recount a similar problem and one of the last suggestions is to check the permissions of ld-linux.so* (It should be noted that first I’ve checked out the /etc/passwd permissions (and contents) as well as /etc/shadow and /etc/group and /etc/gshadow and browsed /etc to see if any files have changed around the time this happened. No luck there, all permissions for those files seem correct and no major file changes so the /lib/ld-linux.so* permissions might be my best bet.
No here’s where I made a BIG mistake with permissions myself…. I found /lib/ld-linux.so.2 -> ld-2.3.3.so and promptly tried chmod 755 ld-linux.so.2 and then by mistake chmod 644 ld-2.3.3.so ….. that was BAD because promptly root had “permission denied” for most everything. Hint…. if you want to hose a system by altering just one file… not even by altering the file, just the permissions – restrict permissions on the /lib/ld* files…. valuable lesson. So, my next step was getting kubuntu to boot from a cd so I could mount the partition and reset the permissions to 755 on the troublesome ld*….. There’s a sidestory here on systems that for some reason can’t boot from a cd, but I’ll save that for another post….
Anyway, root finally able to login again after fixing the lib/ld* permissions that I screwed up…. (Oh, by the way, the system wouldn’t even boot with /lib/ld* at 644….) I was disappointed to find that a regular user still couldn’t log in. Another of the posts that I had found that suggested permissions issues with /lib/ld* was this article at linuxgazette. In it he also mentioned using strace to trace what exactly was happening when the login or su command was taking place. I reviewed the items he had tried (permissions on the home directory /etc/passwd/ and so on…) and then started trying to strace a failed su and a successful su (root.) One thing I hadn’t tried though was emptying utmp and wtmp cat >/var/log/wtmp
cat >var/run/utmp… still no luck.
strace did not seem overly helpful… when doing su username…. I saw several reads of important files /etc/passwd /etc/shadow various pam files with no error, then it would setfsuid32 to my id# and setfsgid32 to my id# with no problems and the very next access to /etc/passwd was permission denied, after a few moments I would see “could not open session” and I’d be back at the root prompt. I started wondering if there was something wrong with /etc/passwd again… I tried out usermod to try and “reset” my uid and shell/username to what they should be…. still “could not open session” I next started looking in the /etc/pam.d directory at the details of each file. I even altered /etc/pam.d/su to make certain authentication items optional instead of required. I was able to get past the “could not open session” message to see /bin/bash permission denied.
At this point I was sure my file permissions on the important files in /etc were good because I’d reviewed them 4 or 5 times, the contents seemed good, the permissions in /lib/ all seemed good…. I was running out of good suggestions or ideas (by the way /home/user permissions were also good.) Finally, this forum thread (scroll about 3/4 of the way down to “problem solved” post) and this post gave another possible answer… DIRECTORY permissions.
You see from the 1st post linked above…. “I remembered the book saying that all directories should be 755 by default, and then we changed two of them so diffeent values. Well, /dev, /mnt, /proc, and /opt weren’t at 755.” Hm…. I looked, /bin and /etc were the first I changed to 755 *(chmod 755 /etc /bin)
Then I tried su and I got a number of errors, but was able to log in…. AHA…. then I went on to chmod 755 /proc /usr/bin /sbin /usr/sbin /var etc…. and the problem was finally solved. Finally, I could su to a regular user or login as a regular user. Several services had to be restarted (because they had try to drop priviliges when they run and essentially lost rights to write a log file, or create a temporary file, etc.
So, how did my directory permissions change? I have not one clue. I had two software installs going, one via urpmi, the other was a source compile and install (razor2). In reviewing the history I don’t see a shell command that could have come close, my best guess is that one of the post install scriptlets in the rpms went astray. That’s my best theory at this point.
The moral of the story is that file permissions are important and the wrong permissions on ONE file (/lib/ld*) can prevent a system from booting. BUT FOLDER permissions are important too. Execute permissions are necessary for a user to even change into a folder and in the case of /bin if you can’t change into it you likely can’t run /bin/bash….. I feel like I should have earned a merit badge of some sort for this one. It reminds me of the FIRST time I had a linux install that I was locked out of (expired passwords) which started me learning more about linux so I wouldn’t have to do a reinstall.