Yikes, what an evening….. it started innocently enough in the afternoon. I have an old Mandrake 10.0 server that I was upgrading clamav on (recent security update). While I was at it, I was reviewing the anti-spam setup to see if I could get any better success with filtering junk mail. spamassassin has had an update since I updated this one last and also it seemed that dcc was installed, but not in active use *(no indications that it’s being used at all.) So, I set about trying to fix that and install the latest spamassassin. Somehere along the way something BAD happened. In retrospect, I’m not entirely sure how, but at one point I was root having just installed the rpms for spamassassin and then exited to my user account. Promptly on switching back to my user account I got a “permission denied” error. Eh? Ok, well let’s su again and see what’s up…. “permission denied” uh oh…. ls “permission denied” most everything actually….. permission denied.
What was worse is that I couldn’t initiate a new login session as a user either. Unable to start session was the last error, but couldn’t cd to /home/username was the first message. I could only log in as root and got a permission denied error for su to any other user. Of course, I spent some time googling and found many examples of similar problems. Unfortunately, there seemed scant solid answers as to what might have happened. (Or good examples of “try this” with successful results.) This groups thread seems to recount a similar problem and one of the last suggestions is to check the permissions of ld-linux.so* (It should be noted that first I’ve checked out the /etc/passwd permissions (and contents) as well as /etc/shadow and /etc/group and /etc/gshadow and browsed /etc to see if any files have changed around the time this happened. No luck there, all permissions for those files seem correct and no major file changes so the /lib/ld-linux.so* permissions might be my best bet.
No here’s where I made a BIG mistake with permissions myself…. I found /lib/ld-linux.so.2 -> ld-2.3.3.so and promptly tried chmod 755 ld-linux.so.2 and then by mistake chmod 644 ld-2.3.3.so ….. that was BAD because promptly root had “permission denied” for most everything. Hint…. if you want to hose a system by altering just one file… not even by altering the file, just the permissions – restrict permissions on the /lib/ld* files…. valuable lesson. So, my next step was getting kubuntu to boot from a cd so I could mount the partition and reset the permissions to 755 on the troublesome ld*….. There’s a sidestory here on systems that for some reason can’t boot from a cd, but I’ll save that for another post….
Anyway, root finally able to login again after fixing the lib/ld* permissions that I screwed up…. (Oh, by the way, the system wouldn’t even boot with /lib/ld* at 644….) I was disappointed to find that a regular user still couldn’t log in. Another of the posts that I had found that suggested permissions issues with /lib/ld* was this article at linuxgazette. In it he also mentioned using strace to trace what exactly was happening when the login or su command was taking place. I reviewed the items he had tried (permissions on the home directory /etc/passwd/ and so on…) and then started trying to strace a failed su and a successful su (root.) One thing I hadn’t tried though was emptying utmp and wtmp cat >/var/log/wtmp
cat >var/run/utmp… still no luck.
strace did not seem overly helpful… when doing su username…. I saw several reads of important files /etc/passwd /etc/shadow various pam files with no error, then it would setfsuid32 to my id# and setfsgid32 to my id# with no problems and the very next access to /etc/passwd was permission denied, after a few moments I would see “could not open session” and I’d be back at the root prompt. I started wondering if there was something wrong with /etc/passwd again… I tried out usermod to try and “reset” my uid and shell/username to what they should be…. still “could not open session” I next started looking in the /etc/pam.d directory at the details of each file. I even altered /etc/pam.d/su to make certain authentication items optional instead of required. I was able to get past the “could not open session” message to see /bin/bash permission denied.
At this point I was sure my file permissions on the important files in /etc were good because I’d reviewed them 4 or 5 times, the contents seemed good, the permissions in /lib/ all seemed good…. I was running out of good suggestions or ideas (by the way /home/user permissions were also good.) Finally, this forum thread (scroll about 3/4 of the way down to “problem solved” post) and this post gave another possible answer… DIRECTORY permissions.
You see from the 1st post linked above…. “I remembered the book saying that all directories should be 755 by default, and then we changed two of them so diffeent values. Well, /dev, /mnt, /proc, and /opt weren’t at 755.” Hm…. I looked, /bin and /etc were the first I changed to 755 *(chmod 755 /etc /bin)
Then I tried su and I got a number of errors, but was able to log in…. AHA…. then I went on to chmod 755 /proc /usr/bin /sbin /usr/sbin /var etc…. and the problem was finally solved. Finally, I could su to a regular user or login as a regular user. Several services had to be restarted (because they had try to drop priviliges when they run and essentially lost rights to write a log file, or create a temporary file, etc.
So, how did my directory permissions change? I have not one clue. I had two software installs going, one via urpmi, the other was a source compile and install (razor2). In reviewing the history I don’t see a shell command that could have come close, my best guess is that one of the post install scriptlets in the rpms went astray. That’s my best theory at this point.
The moral of the story is that file permissions are important and the wrong permissions on ONE file (/lib/ld*) can prevent a system from booting. BUT FOLDER permissions are important too. Execute permissions are necessary for a user to even change into a folder and in the case of /bin if you can’t change into it you likely can’t run /bin/bash….. I feel like I should have earned a merit badge of some sort for this one. It reminds me of the FIRST time I had a linux install that I was locked out of (expired passwords) which started me learning more about linux so I wouldn’t have to do a reinstall.
Related PostsRelated Posts
- Denyhosts as an added defence to ssh server A couple days ago I had a brief article on the vandals banging away at the door of my ssh server. Like I said, I've, at times, been fairly smug abou the futility of their actions, but.... the persistance concerns me. Let me be more specific, I keep a fairly......
- Glide Effortless to be the first "browser as OS" Ok - browser as OS is a term that has been thrown about and speculated on. The concept is to provide a set of applications through ANY web browser such that it's suitable for doing the majority of your work. Most people speculate on Google being the company to bring......
- Sleuthkit - windows and linux file recovery http://www.sleuthkit.org/ Sluethkit... is a collection of tools for forensic analysis of a system. Usually it's something that would be done when you've had a suspected rootkit on the system and you boot to another operating system with sluethkit installed (maybe livecd/etc.) and want to try to analyze and hunt for......
- Are Free Registry Cleaners Really Safe to Use? When you use a computer for some amount of time it becomes slow. This is nothing new, every computer user experiences this sooner or later. After some recommendations from a friend I tried to install a free registry cleaner. It all looked all smooth and nice when I ran it.......
- Mac Keeper document.write(''); What Makes MacKeeper Stand Out? MacKeeper’s utilities, tools and services will keep your Mac: Increase Mac PerformanceAward Winning app for top performance of your Mac.document.write(''); Clean By protecting your hard drive from junk and other useless data. Secured By hiding, encrypting and backing up any important files. Reliable......
- Wordpress Backup Wordpress Backup is an essential plugin for all Wordpress blog administrators by the Blog Traffic Exchange. It performs regular backups of your upload (images) current theme, and plugin directories. Backup files are available for download and optionally emailed to a specified email. Don't get caught without a recent backup of......
- Administrative access on linux systems
- Adding swap space to a running linux system
- Apple Mac OS X Not Accepting Login
- Running windows applications directly in linux
- Outlook archive error message