System software
A Solution for Mosh Scrollback
Mosh is a pretty good tool, almost indispensable when working in places with crappy internet. While it is designed to help with situations like “LTE on the beach,” it actually works very well in places where internet connectivity is genuinely bad: 1500msec RT, latency, 30% packet loss, and frequent drops in connectivity that last seconds to hours, otherwise known as most of the world. On a good day I lose an SSH connection randomly about every 3-6 hours but I’ve only ever lost a Mosh session when my system went down.
It does a lot of things, but two are key for my use: it syncs user input in the background while local echoing what you type so you can finish your command (and correct a typo) without waiting 1500msec for the remote echo to update; and it creates persistent connections that survive drop off of almost any type except killing the terminal application on one end or the other (anything between can die and when it recovers, you catch up). This means compiles finish and you actually get the output warnings…
…well…
…some of them. Because Mosh’s one giant, glaring, painful, almost debilitating weakness is that it doesn’t support scrollback. So compared to tmux or something else that you can reconnect to after your SSH session drops, you really lose screen content, which is a PITA when ls-ing a directory. I mean, it isn’t that much of an efficiency gain to have to type “ls | less” instead of just “ls” every time you want to see a directory.
I found a solution that works for me. I also use Tmux with Mosh because Tmux will survive a dead client and working with Windows client reboots are a fact of life (I know, sad, but there are some tools I still need on windows, hopefully not for much longer).
Tmux has a facility for creating a local log file, which I then “tail -f” using a separate SSH window. If the SSH client disconnects, no loss, I can pick up the log anytime. It is just mirroring everything that the mosh terminal is doing and the scroll bar scroll back works fine. And it is a raw text file, so you can pipe the output through grep to limit what’s displayed to something of interest and review the log asynchronously as, say, a build is progressing.
Although there are some nice advantages to this, when/if Mosh supports scrollback, it’ll be far more convenient having it in the same window, but for now this is the easiest solution I could come up with.
FreeBSD:
# portmaster sysutils/tmux # portmaster net/mosh # ee ~/.tmux.conf -> bind-key H pipe-pane -o "exec cat >>$HOME/'#W-tmux.log'" \; display-message 'Logging enabled to $HOME/#W-tmux.log' -> set -g history-limit 30000 Start a Mosh session (for example with
on windows) # tmux # [CTRL]-b H start SSH session (Mobaxterm or
on windows)
# tail -f csh-tmux.log
("csh" will be the name of the mosh window - so really "(MoshWindowName)-tmux.log"
You can tmux the ssh session too and still have scrollback and then just reconnect into the same tail command, which preserves the whole scrollback. If you’re on a connection like I’m on, your scrollback logfile will drop off a couple of times a day, but you won’t lose your Mosh session, and it’ll be waiting for you when you’re reminded that you need to see those security warnings from the compile that just scrolled off the Mosh screen forever.
cyrus-sasl-saslauthd-2.1.26 auth_krb5.c compile error
Upgrading cyrus-sasl-saslauthd-2.1.25 to the current cyrus-sasl-saslauthd-2.1.26, I started to get auth_krb5.c compile errors that were terminating the compile like:
<command-line>: warning: this is the location of the previous definition
mv -f .deps/auth_getpwent.Tpo .deps/auth_getpwent.Po
cc -DHAVE_CONFIG_H
-DSASLAUTHD_CONF_FILE_DEFAULT=\"/usr/local/etc/saslauthd.conf\" -I. -I.
-I.. -I. -I./include -I./include -I./../include -I/usr/local/include
-DKRB5_HEIMDAL -I/usr/local/include -O3 -pipe -march=native
-DLDAP_DEPRECATED -fno-strict-aliasing -MT auth_krb5.o -MD -MP -MF
.deps/auth_krb5.Tpo -c -o auth_krb5.o auth_krb5.c
In file included from mechanisms.h:35,
from auth_krb5.c:51:
saslauthd.h:190:1: warning: "KRB5_HEIMDAL" redefined
<command-line>: warning: this is the location of the previous definition
auth_krb5.c: In function 'auth_krb5_init':
auth_krb5.c:105: warning: assignment discards qualifiers from pointer
target type
auth_krb5.c:106: warning: assignment discards qualifiers from pointer
target type
auth_krb5.c: In function 'auth_krb5':
auth_krb5.c:184: error: 'krb5_verify_opt' undeclared (first use in this
function)
auth_krb5.c:184: error: (Each undeclared identifier is reported only once
auth_krb5.c:184: error: for each function it appears in.)
auth_krb5.c:184: error: expected ';' before 'opt'
auth_krb5.c:233: error: 'opt' undeclared (first use in this function)
*** Error code 1
Stop in
/usr/ports/security/cyrus-sasl2-saslauthd/work/cyrus-sasl-2.1.26/saslauthd.
*** Error code 1
Stop in
/usr/ports/security/cyrus-sasl2-saslauthd/work/cyrus-sasl-2.1.26/saslauthd.
*** Error code 1
Stop in /usr/ports/security/cyrus-sasl2-saslauthd.
with some expert advice from the port maintainer, Hajimu UMEMOTO (what is not to love about BSD and open source? Something goes wrong, the guy who knows everything about it tells you how to fix it right away). He correctly ascertained that I had security/krb5 installed, a dependency of openssh-portable. Kerberos, HEIMDAL and GSSAPI occasionally have interactions, but his advice was to make with the directive KRB5_HOME=/usr/local. I put this into /etc/make.conf to make it permanent, deinstall/reinstalled security/krb5 and then cyrus-sasl-2.1.26 compiled perfectly.
Thanks Mr Umemoto!
Lenovo System Update breaks Windows Update
An unfortunate series of events afflicted my poor Lenovo W500. At some point I started to get odd errors and ran sfc /scannow and found a large set of uncorrectable errors in a variety of packages. Nothing caused me too much trouble, so I ignored it. I kept hoping some giant windows update would overwrite all the broken bits and save me the trouble of debugging it, so I was happy when Win 7 Service Pack 1 was finally available – at 70-400MB it has to overwrite just about everything, but my happiness was short lived.
Sadness… somewhere in the preamble updates something got hosed and a check of my disk showed bad blocks. Chkdisk confirmed it and it seemed a failing disk was likely the cause of many of my woes. I strapped as many belts and suspenders around the disk as I could – windows backup, clonezilla, copying files. Clonezilla couldn’t read all the blocks, so I had to use the recover option, but that version still had problems. Dang.
Windows recovery was fail, rollback, in place upgrade, system restore. All fail. Fine. Life sucks – reinstall from scratch and then reinstall all my applications. This is a huge pain in the ass, but windows just get sluggish in a year or so without a complete reinstall anyway; it isn’t like Microsoft cares whether you can get your work done or not, what are you going to do? Pay 100% style premium so The Steve can dictate what you can do? When choosing one evil empire over another, pick the cheapest.
So I do a reinstall from scratch. Windows reinstalls more than a few weeks out from the release of the OS are a monumental undertaking as the updates take forever. Bringing a windows 7 computer up to date takes between 1-1.5GB of updates, after installing a DVD’s worth of software. There’s the endless reboots as patches are installed and removed and whatever, multi-hour downloads. But eventually, you get a perfect, up-to-date OEM blessed configuration. Or so you think… duh duh duh.
I finished the whole mess, including the Lenovo System Update drivers and windows update stopped working and sfc /scannow gave me errors. Crappenfest. Reverting to the first system snapshot failed, uninstalling every single thing – all windows updates, all Lenovo updates was fail. Whatever did this can’t be fixed after it is done. You’re screwed.
Nothing to do but try again from scratch, this time paying attention and not using the computer at all until everything was installed, including anti-virus. Another 36 hours of updates later, same result. CRAPPENFEST.
How could that be? Some OEM/M$ update is breaking the system, and so began the hunt: reinstall from scratch #3. I used a binary search algo, saving disk images between each iteration so I wouldn’t have to do install from scratch 4. All windows update updates were fine, so the problem was with Lenovo. Updating only essential components was fail, restore windows. Installing just the really important bits one or two at a time (not quite binary splitting the install batch) got me through about half the useful lenovo updates, so time to create an image.
Reviewing the Device Manager, I saw 5 “?” devices – and searching around I found they were related to 3 drivers:
4-in-1 Card reader
Setup from “4in1” folder
- Base system device Ricoh Memory Stick controller
- Base system device Ricoh SD/MMC host controller
- Base system device Ricoh XD- picture card controller
the above 3 unknown devices in device manager will be resolved by the Ricoh cardreader drivers.
Turbo Memory
Some models may or may not have a turbo memory module.
To verify, open “device manager” and check for a unknown device listed as “PCI Memory Controller”.
If such a device is listed, then install the Turbo Memory driver via setup from “turbomem” folder.
You may see a hardware device install popup from systray.
Reboot is required.
AMT
If Intel’s Adaptive Management Technology is implemented in anenterprise enviroment, then the AMT drivers can be installed viasetup from the “AMT\MEI” folder.
If AMT is not employed, AMT can be disabled via Bios.
The drivers from the AMT setup will resolve the unknown devices within “device mananger”, the PCI Simple Communications Controller.
I install just those.
FAIL!
recover using windows system recovery tools (format disk, reinstall from image)
One of those three. AMT I don’t want anyway, so I tried to disable it in BIOS, but there were no entries in my BIOS for AMT so I just disabled it in device manager. ?->! np. I don’t trust it, too deep in the OS. This seems like the problem.
Install Ricoh, seems benign. Reboot, system works. Just Turbo Memory to go.
Turbo Memory is kind of cool, especially for a laptop. As Intel says:
The benefits of Intel Turbo Memory include:
- Faster application load and run time when multi-tasking
- Faster boot time
- Lowers PC power consumption by reducing hard drive spin
It uses some special on the mobo Intel cache memory to speed up disk access like a hybrid SSD/Rotating disk. I want this to work ’cause it cost money to put in the computer. Since it had to be that evil AMT security thing, no problem. FAIL. The problem is Intel’s Turbo Memory driver. If you install it, you’re screwed. Now that I know what the problem is, I find I’m not the only one with it.
I tried both the Lenovo supplied Turbo Memory Driver and the Intel supplied one here . Both are fail. No Turbo Memory For You. There are some hints in forums that maybe Turbo Memory isn’t compatible with advanced format disk drives, so possibly replacing my older 500GB disk with a newer 700GB uncovered a latent incompatibility.
Given how much of a disaster installing it is–the only recovery method is to restore a previous disk image–I suppose that’s one feature of my MoBo that is obsolete now. Bummer. Ate just about a week of work time to find this little monster of a driver. Thanks WinTel.
(my sfc /scannow log was filled with entries like:
POQ 119 starts: 0: Move File: Source = [l:192{96}]"\SystemRoot\WinSxS\Temp\PendingRenames\a52557019366cc01d63500006c0a3c08._0000000000000000.cdf-ms", Destination = [l:104{52}]"\SystemRoot\WinSxS\FileMaps\_0000000000000000.cdf-ms" 1: Move File: Source = [l:162{81}]"\SystemRoot\WinSxS\Temp\PendingRenames58759019366cc01d73500006c0a3c08.$$.cdf-ms", Destination = [l:74{37}]"\SystemRoot\WinSxS\FileMaps\$$.cdf-ms" 2: Move File: Source = [l:224{112}]"\SystemRoot\WinSxS\Temp\PendingRenames58759019366cc01d83500006c0a3c08.$$_microsoft.net_3296b36dbe4c7fa3.cdf-ms", Destination = [l:136{68}]"\SystemRoot\WinSxS\FileMaps\$$_microsoft.net_3296b36dbe4c7fa3.cdf-ms" 3: Move File: Source = [l:244{122}]"\SystemRoot\WinSxS\Temp\PendingRenames\c6495e019366cc01d93500006c0a3c08.$$_microsoft.net_framework_83386eac0379231b.cdf-ms", Destination = [l:156{78}]"\SystemRoot\WinSxS\FileMaps\$$_microsoft.net_framework_83386eac0379231b.cdf-ms" 4: Move File: Source = [l:266{133}]"\SystemRoot\WinSxS\Temp\PendingRenames\26ab60019366cc01da3500006c0a3c08.$$_microsoft.net_framework_v2.0.50727_e9368840261e60ee.cdf-ms", Destination = [l:178{89}]"\SystemRoot\WinSxS\FileMaps\$$_microsoft.net_framework_v2.0.50727_e9368840261e60ee.cdf-ms" 5: Move File: Source = [l:288{144}]"\SystemRoot\WinSxS\Temp\PendingRenames\860c63019366cc01db3500006c0a3c08.$$_microsoft.net_framework_v2.0.50727_redistlist_2e6ab8b35e9ef953.cdf-ms", Destination = [l:200{100}]"\SystemRoot\WinSxS\FileMaps\$$_microsoft.net_framework_v2.0.50727_redistlist_2e6ab8b35e9ef953.cdf-ms"
POQ 119 ends.
Rsync corrupted MAC on input
I am migrating my FreeNAS 7.x to a 8.x, which means copying the ZFS tank as there isn’t a tool for migrating the disks right now and upgrading them to the version of ZFS in 8.x. Kind of a pain in the butt that was made worse by the endless recurrence of an error like:
Received disconnect from xxx.xxx.xxx.xxx: 2: Packet corrupt
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (23734 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]
or something like:
Disconnecting: Packet corrupt
rsync: connection unexpectedly closed (581052724 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [receiver=3.0.8]
rsync: connection unexpectedly closed (202 bytes received so far) [generator]
rsync error: unexplained error (code 255) at io.c(601) [generator=3.0.8]
I figured my 7.x install had to be fine as I’ve been RSYNCing my server to it without error for about a year now, so the problem had to be in the new box and poking around for “packet corrupt rsync” on google was turning up a lot of *shrug* maybe bad RAM or a bad NIC. Hmmm… I tried command line push and pull from both boxes via SSH to see if I could get better results, no luck: a few files would transfer, maybe 10 seconds, maybe 5 minutes, then blop, bad packet, broken pipe, oh so informative “unexplained error” at io.c, start over. No way I was going to be able to transfer 3.5 TB 100MB at a time.
Finally I found this and checked the lovely graphical status monitor on the FreeNAS 7 box. It has 4GB of RAM, whichhas been plenty so far, but looking at the graph it was using about 95% of that memory. It had been up for 59 days so I was reluctant to reboot it, I mean uptime is a competition after all. But I took a dive and rebooted. Now, even with CIFS/SAMBA cranking some backup files simultaneously, RSYNC is running flawlessly at a nice steady 300mb/s, apparently limited by CPU (seems to be single threaded, maxing out one CPU and leaving the other idle, hmmm… problem for another day). I feel bad for doubting my FreeNAS 8 box, it was never the problem.
So if you’re getting RSYNC problems consider rebooting the server to free up RAM or even upgrading. The new box will have 12-16GB, which is about what is recommended for ZFS (1GB/TB) and things are looking pretty good. My RSYNC was running just -a –progress, no resource intensive -z option.