About a week ago I had a sort of system crash on my Intrepid server. After performing some rote maintenance, and deploying a vanity Drupal site for my daughter I was getting the “permission denied” error from Apache on all of my domains and sub-domains. To top this off, no users I host mail for on that machine (my daughter, a bot, and myself) were capable of receiving any mail. Great. After making sure permissions and ownerships were correct on my web folders (where I had to fix some strange anomalies), scouring the configuration settings for apache (which all seemed correct), and my separate sites, Drupal and otherwise, (again, no issues), I checked my mail logs (same machine) and found an interesting set of errors.
fatal: get_service_attr: unknown username: vmail
warning: private/dovecot socket: malformed response
warning: transport dovecot failure -- see a previous warning/fatal/panic logfile record for the problem description
warning: process /usr/lib/postfix/pipe pid 8711 exit status 1
warning: /usr/lib/postfix/pipe: bad command startup -- throttling
All from postfix. Great. A quick trip into /etc/passwd
told me that vmail
was in fact a user…but the user name for all of my web stuff was not in the list.?. Added the user and presto! My sites come back. I did some google-ing of those postfix errors, but nothing too productive came back, and I even went into the #postfix room on freenode, and was told to make sure that the service referred to in those errors was not in chroot
status. Duh. It was not, so I actually purged, reinstalled, and reconfigured my mail-server. Same errors. Reboot.?. “Let’s ssh back into the system” I thought. There is some permission issue that is keeping mail delivery from happening and killed my webserver. All I did in my “rote maintenance” was remove a user and group from my system that was a leftover from an installation of a server management system I tried out a few months back that I didn’t like…well, when I ssh’ed in, I saw the darnedest thing. In the terminal I didn’t see my usual
joshp@host:$
.
Instead i saw
I have no name!@host:$
My head did not explode. Instead it urged me to google that. I was able to log in with my user name and password, but apparently not have a name after logging in. A quick search led me to this post in the Ubuntu Forums. I followed the advice in that post, and checked the permissions on /ect/passwd
and sure enough, something had altered them. I fixed this, and suddenly, as light of hope from the heavens, my entire system went back to normal. Logged out and in via ssh to the happy joshp@host$
that I was used to, mail was being delivered to mailboxes, and my Drupal, and other sites, were behaving again. I was able to get the system, and my daughters vanity site, and email set up just in time for her to get to it for a school project. Win! Lessons?
- If you get locked out of all of your own sites, can’t receive mail suddenly, or see
I have no name!@host$
in your terminal, check the permissions on your/etc/passwd
file first. - Be weary of all-in-one server management solutions that take over and rearrange your server setup for the promise of easy administration. I had one apparently change the permissions on a pretty systems critical file, and try to take over management of all system permissions. Being only 5 9’s certain of this and not 100%, I am not offering the name of this package, but watch what you install, and watch what is left behind when you uninstall! SSH is a fine remote login option, why reinvent the wheel?
- Ubuntu Forums is an excellent place to find problems to your server-related headaches! I always associated those forums with desktop use, but more and more find them an invaluable resource for server related issues.
Thank you Ubuntu Forums for helping me dispell yet another evil.
Amusing story – I would have panicked seriously at the “I have no name!” prompt, thinking an intruder came in.
You might be interested in etckeeper (should be in the universe repository). It keeps checks on your /etc through a version control system (I use it with bzr, but it supports several systems), and can not only track line-by-line file changes but also file permissions.
Awesome…I will be looking into that tonight.