Damn, I tried to upgrade FreeBSD remotly on one web server I admin and I've managed to broke authorization in ssh. It is still up but I cannot login. Neither using my key nor using my password. The only way I can login on server is FTP. What can I do with it? Should I try to edit my crontab to start something I can use as remote shell? Or should I try to put CGI - shell simulator?
This accident leads me to idea that I should turn on telnet service during upgrades as backup remote shell service. Were I did it telnet could save my ass.
I had that one bite me once, too - luckily I still had one of my ssh sessions connected. The key is to not do the installkernel and installworld steps when you're going to be disconnecting from the system. You can toss the buildworld and buildkernel into a screen and forget about them, but when you come back to do the install steps and mergemaster, make certain that you complete the process and reboot!
Most times ssh doesn't change significantly, but when it does, you get in trouble with things like this. Your best bet is probably to hack yourself a shell somehow... one of those times when having good security is bad. If the server isn't too vital, you could just reboot it and clean up the mess from there - though if you haven't run mergemaster yet and you're upgrading a significant distance, you could have some nasty problems.
Good luck!
Re:SSH fun
IlyaM on 2002-08-07T16:59:04
In ideal world nobody should do system upgrades remotly: i.e. you should be able to access system console and should do everything in single user. *sigh*I found SSH to be broken after I had ran mergemaster. I suspect that I messed with PAM settings. I did touched SSH configs too but it should not affect SSH daemon as I haven't restarted it.
Luckily I did run installworld before logging off and I have very slim hope that newly installed SSH will be compatible with new configs. I'll find it tomorrow when I'll get somebody with physical access to this box to reboot it.
Re:SSH fun
krellis on 2002-08-07T17:30:44
I do remote system upgrades on a regular (read almost daily) basis - if you practice enough with non-critical systems and get a good, functional procedure down, it can go off flawlessly. On some systems (pretty much everything I have in production, for example) you simply can't afford the amount of downtime that going down into single user even for just a couple of quick reboots and installs will generate.
If you ran installworld and mergemaster before logging off, you should be good to go when you reboot - the problem is simply that your sshd binary and configs are for openssh 3.4 while your running daemon is 2.9 (with FreeBSD-specific patches).
Re:SSH fun
IlyaM on 2002-08-07T18:53:23
I've been reading freebsd maillists for quite long time and I've seen remote upgrades discussed many times. In short conclusion I've drawn from these discussions is that strictly speaking non-single user upgrades are not supported and in general bad thing. There are many reasons why. Risk of broken networking, risk of unbootable system, undefined state of systen which may lead to security compromise. If you use security levels you will have to disable them and reboot without them into multy-user before doing upgrade. This again increases risks of security compromises. Having said this I must admit that I do remote upgrades from time to time and in reality I have no choice.
