System software
A Solution for Mosh Scrollback
Mosh is a pretty good tool, almost indispensable when working in places with crappy internet. While it is designed to help with situations like “LTE on the beach,” it actually works very well in places where internet connectivity is genuinely bad: 1500msec RT, latency, 30% packet loss, and frequent drops in connectivity that last seconds to hours, otherwise known as most of the world. On a good day I lose an SSH connection randomly about every 3-6 hours but I’ve only ever lost a Mosh session when my system went down.
It does a lot of things, but two are key for my use: it syncs user input in the background while local echoing what you type so you can finish your command (and correct a typo) without waiting 1500msec for the remote echo to update; and it creates persistent connections that survive drop off of almost any type except killing the terminal application on one end or the other (anything between can die and when it recovers, you catch up). This means compiles finish and you actually get the output warnings…
…well…
…some of them. Because Mosh’s one giant, glaring, painful, almost debilitating weakness is that it doesn’t support scrollback. So compared to tmux or something else that you can reconnect to after your SSH session drops, you really lose screen content, which is a PITA when ls-ing a directory. I mean, it isn’t that much of an efficiency gain to have to type “ls | less” instead of just “ls” every time you want to see a directory.
I found a solution that works for me. I also use Tmux with Mosh because Tmux will survive a dead client and working with Windows client reboots are a fact of life (I know, sad, but there are some tools I still need on windows, hopefully not for much longer).
Tmux has a facility for creating a local log file, which I then “tail -f” using a separate SSH window. If the SSH client disconnects, no loss, I can pick up the log anytime. It is just mirroring everything that the mosh terminal is doing and the scroll bar scroll back works fine. And it is a raw text file, so you can pipe the output through grep to limit what’s displayed to something of interest and review the log asynchronously as, say, a build is progressing.
Although there are some nice advantages to this, when/if Mosh supports scrollback, it’ll be far more convenient having it in the same window, but for now this is the easiest solution I could come up with.
FreeBSD:
# portmaster sysutils/tmux # portmaster net/mosh # ee ~/.tmux.conf -> bind-key H pipe-pane -o "exec cat >>$HOME/'#W-tmux.log'" \; display-message 'Logging enabled to $HOME/#W-tmux.log' -> set -g history-limit 30000 Start a Mosh session (for example with
on windows) # tmux # [CTRL]-b H start SSH session (Mobaxterm or
on windows) # tail -f csh-tmux.log ("csh" will be the name of the mosh window - so really "(MoshWindowName)-tmux.log"
You can tmux the ssh session too and still have scrollback and then just reconnect into the same tail command, which preserves the whole scrollback. If you’re on a connection like I’m on, your scrollback logfile will drop off a couple of times a day, but you won’t lose your Mosh session, and it’ll be waiting for you when you’re reminded that you need to see those security warnings from the compile that just scrolled off the Mosh screen forever.
cyrus-sasl-saslauthd-2.1.26 auth_krb5.c compile error
Upgrading cyrus-sasl-saslauthd-2.1.25 to the current cyrus-sasl-saslauthd-2.1.26, I started to get auth_krb5.c compile errors that were terminating the compile like:
<command-line>: warning: this is the location of the previous definition mv -f .deps/auth_getpwent.Tpo .deps/auth_getpwent.Po cc -DHAVE_CONFIG_H -DSASLAUTHD_CONF_FILE_DEFAULT=\"/usr/local/etc/saslauthd.conf\" -I. -I. -I.. -I. -I./include -I./include -I./../include -I/usr/local/include -DKRB5_HEIMDAL -I/usr/local/include -O3 -pipe -march=native -DLDAP_DEPRECATED -fno-strict-aliasing -MT auth_krb5.o -MD -MP -MF .deps/auth_krb5.Tpo -c -o auth_krb5.o auth_krb5.c In file included from mechanisms.h:35, from auth_krb5.c:51: saslauthd.h:190:1: warning: "KRB5_HEIMDAL" redefined <command-line>: warning: this is the location of the previous definition auth_krb5.c: In function 'auth_krb5_init': auth_krb5.c:105: warning: assignment discards qualifiers from pointer target type auth_krb5.c:106: warning: assignment discards qualifiers from pointer target type auth_krb5.c: In function 'auth_krb5': auth_krb5.c:184: error: 'krb5_verify_opt' undeclared (first use in this function) auth_krb5.c:184: error: (Each undeclared identifier is reported only once auth_krb5.c:184: error: for each function it appears in.) auth_krb5.c:184: error: expected ';' before 'opt' auth_krb5.c:233: error: 'opt' undeclared (first use in this function) *** Error code 1 Stop in /usr/ports/security/cyrus-sasl2-saslauthd/work/cyrus-sasl-2.1.26/saslauthd. *** Error code 1 Stop in /usr/ports/security/cyrus-sasl2-saslauthd/work/cyrus-sasl-2.1.26/saslauthd. *** Error code 1 Stop in /usr/ports/security/cyrus-sasl2-saslauthd.
with some expert advice from the port maintainer, Hajimu UMEMOTO (what is not to love about BSD and open source? Something goes wrong, the guy who knows everything about it tells you how to fix it right away). He correctly ascertained that I had security/krb5
installed, a dependency of openssh-portable
. Kerberos, HEIMDAL and GSSAPI occasionally have interactions, but his advice was to make with the directive KRB5_HOME=/usr/local
. I put this into /etc/make.conf
to make it permanent, deinstall/reinstalled security/krb5
and then cyrus-sasl-2.1.26
compiled perfectly.
Thanks Mr Umemoto!
Rsync corrupted MAC on input
I am migrating my FreeNAS 7.x to a 8.x, which means copying the ZFS tank as there isn’t a tool for migrating the disks right now and upgrading them to the version of ZFS in 8.x. Kind of a pain in the butt that was made worse by the endless recurrence of an error like:
Received disconnect from xxx.xxx.xxx.xxx: 2: Packet corrupt
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (23734 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]
or something like:
Disconnecting: Packet corrupt
rsync: connection unexpectedly closed (581052724 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [receiver=3.0.8]
rsync: connection unexpectedly closed (202 bytes received so far) [generator]
rsync error: unexplained error (code 255) at io.c(601) [generator=3.0.8]
I figured my 7.x install had to be fine as I’ve been RSYNCing my server to it without error for about a year now, so the problem had to be in the new box and poking around for “packet corrupt rsync” on google was turning up a lot of *shrug* maybe bad RAM or a bad NIC. Hmmm… I tried command line push and pull from both boxes via SSH to see if I could get better results, no luck: a few files would transfer, maybe 10 seconds, maybe 5 minutes, then blop, bad packet, broken pipe, oh so informative “unexplained error” at io.c, start over. No way I was going to be able to transfer 3.5 TB 100MB at a time.
Finally I found this and checked the lovely graphical status monitor on the FreeNAS 7 box. It has 4GB of RAM, whichhas been plenty so far, but looking at the graph it was using about 95% of that memory. It had been up for 59 days so I was reluctant to reboot it, I mean uptime is a competition after all. But I took a dive and rebooted. Now, even with CIFS/SAMBA cranking some backup files simultaneously, RSYNC is running flawlessly at a nice steady 300mb/s, apparently limited by CPU (seems to be single threaded, maxing out one CPU and leaving the other idle, hmmm… problem for another day). I feel bad for doubting my FreeNAS 8 box, it was never the problem.
So if you’re getting RSYNC problems consider rebooting the server to free up RAM or even upgrading. The new box will have 12-16GB, which is about what is recommended for ZFS (1GB/TB) and things are looking pretty good. My RSYNC was running just -a –progress, no resource intensive -z option.