RAM

Rsync corrupted MAC on input

Saturday, August 27, 2011 

I am migrating my FreeNAS 7.x to a 8.x, which means copying the ZFS tank as there isn’t a tool for migrating the disks right now and upgrading them to the version of ZFS in 8.x. Kind of a pain in the butt that was made worse by the endless recurrence of an error like:

Received disconnect from xxx.xxx.xxx.xxx: 2: Packet corrupt
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (23734 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]

or something like:

Disconnecting: Packet corrupt
rsync: connection unexpectedly closed (581052724 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [receiver=3.0.8]
rsync: connection unexpectedly closed (202 bytes received so far) [generator]
rsync error: unexplained error (code 255) at io.c(601) [generator=3.0.8]

I figured my 7.x install had to be fine as I’ve been RSYNCing my server to it without error for about a year now, so the problem had to be in the new box and poking around for “packet corrupt rsync” on google was turning up a lot of *shrug* maybe bad RAM or a bad NIC. Hmmm… I tried command line push and pull from both boxes via SSH to see if I could get better results, no luck: a few files would transfer, maybe 10 seconds, maybe 5 minutes, then blop, bad packet, broken pipe, oh so informative “unexplained error” at io.c, start over. No way I was going to be able to transfer 3.5 TB 100MB at a time.

Finally I found this and checked the lovely graphical status monitor on the FreeNAS 7 box. It has 4GB of RAM, whichhas been plenty so far, but looking at the graph it was using about 95% of that memory. It had been up for 59 days so I was reluctant to reboot it, I mean uptime is a competition after all. But I took a dive and rebooted. Now, even with CIFS/SAMBA cranking some backup files simultaneously, RSYNC is running flawlessly at a nice steady 300mb/s, apparently limited by CPU (seems to be single threaded, maxing out one CPU and leaving the other idle, hmmm… problem for another day). I feel bad for doubting my FreeNAS 8 box, it was never the problem.

So if you’re getting RSYNC problems consider rebooting the server to free up RAM or even upgrading. The new box will have 12-16GB, which is about what is recommended for ZFS (1GB/TB) and things are looking pretty good. My RSYNC was running just -a –progress, no resource intensive -z option.

Posted at 00:24:53 GMT-0700

Category: FreeBSDTechnology

Radioactive Mudworms (1991)

Monday, May 23, 2011 

I animated Radioactive Mudworms in 1991 with a program called Infini-D.  The soundtrack was courtesy of David Lenat. It was first published on the QuickTime Beta CD to Apple Developers and then in 1992 re-rendered on a Mac IIfx 40mhz 68040 with a massive 16MB of RAM in this version for the FigTime commercial CD. As I remember it, this took about a week to render on that massive machine.  I’m pretty sure I ray-traced it, but I output to “thousands” of colors as required by the CODEC and so it is hard to see some of the details.

The file is so old that the “animation” CODEC (RLE) used is no longer supported.  I had to boot my old Mac 8600 to read the CD and convert the file to uncompressed, so I could re-compress it with a modern version of QuickTime.  I was greeted with an alert that my last backup was in 2003.  Time flies, but the mac still runs and that OS 9 operating system is still a nostalgic pleasure.  I used it regularly from 1987-ish to 2003-ish, and it is still the OS I’ve spent the most hours in front of.

Digital obsolescence is starting to consume my work history as the past has already eaten the DECstation streaming tapes my MIT work was “archived” on.  Of course, I can still read my preschool notebooks and I’m sure I could still read my parents notebooks.

Infini-D was my favorite 3D program of the time, though it was supplanted by StrataStudio 3D, Turbo-3D, and finally ProEngineer. It had a nice combination of modelling, rendering, and animation tools and was part of a brief “golden era” of 3D most remarkable for VPL and the existential excitement around Virtual Reality, which comes and goes every decade or so as a new generation reads Snow Crash and thinks that they, too, could be Hero Protagonist or Raven and have a chance with someone like YT if only the world truly adopted VR.

I was reminded of Radioactive Mudworms as I spent the weekend trying to teach the basics of video compression remotely to some coworkers who may not have been born when I made this.

The video was encoded at Valley Green 6, in the cube farm for the Advanced Technology Group at Apple.

A very old .mov version: https://gessel.blackrosetech.com/wp-content/uploads/2011/05/Radioactive-Mudworms.mov

And because technology, here’s an AV1.webm version for the next 32 years given that VP9 may be replaced once Apple gets on board, if they do.

 

BTW, the conversion is

ffmpeg -i Radioactive-Mudworms.mov -vf scale=-1:250:flags=lanczos  -c:v libsvtav1 -pix_fmt yuv420p10le -preset 0 -svtav1-params tune=0 -b:v 0 -crf 22 -pass 1 -an -f null /dev/null && \
 ffmpeg -i Radioactive-Mudworms.mov -vf scale=-1:250:flags=lanczos -c:v libsvtav1 -pix_fmt yuv420p10le -preset 0 -svtav1-params tune=0 -b:v 0 -crf 22 -pass 2 -c:a libopus Radioactive-Mudworms-AV1.webm

 

Posted at 04:21:18 GMT-0700

Category: CodeTechnologyVanity sitesvideo

Moar Privacy

Thursday, December 9, 2010 

I’m using an Ubuntu VM for private browsing, and like many people, I’m stuck using a mainstream OS for much of my work (Win7) due to software availability constraints. But some software works much better in a linux environment and Ubuntu is as pretty as OSX, free, and installs easily on generic x86 hardware.

It is also pretty straightforward to install an isolated and secure browsing instance using VirtualBox. It takes about 20G of hard disk and will use up at least 512K (better 1G) of your system RAM. If you want to run this sort of config, your laptop should have more than enough disk space and RAM to support the extra load without bogging, but it is a very solid solution.

Installing Ubuntu is easy – even easier with an application like VirtualBox – just install virtualbox, download the latest ubuntu ISO, and install from there. If you’re on bare metal, the easiest thing to do is burn a CD and install off that.

Ubuntu desktop comes with Firefox in the tool bar. Customizing for private browsing is a bit more involved.

My first steps are to install:

NoScript is an easy win. It is a bit of a pain to set up at first, but soon you add exceptions for all your favorite sites and while that isn’t great security practice, it is essential for sane browsing. NoScript is particularly helpful when browsing the wacky parts of the net and not getting exotic browsing diseases: it is your default dental dam. Be careful of allowing domains you don’t recognize – Google them first and make sure you understand why they need to run a script on your computer and that it is safe. A lot of sites use partners for things like video feeds, so if some function seems broken, you probably need to allow that particular domain. On the other hand, most of the off-site scripts are tracking or stats and you really don’t need to play along with them.

BetterPrivacy is a new one for me. I am very impressed that it found approximately 1.3 zillion (OK 266) different company flash cookies AFTER I had installed TACO and noscript etc. You bastards. I’m sure I can enjoy hulu without making my play history shared-available to every flash site I might visit. Always Sunny in Philadelphia marks me as a miscreant. I flush the flash cookies on starting silently (preferences).

TACO is a bit intrusive, but it seems to work to selectively block tracking and advertising cookies. At least the pop up is comforting. For private browsing, I’d set it to reject all classes of tracking cookies (change the preferences from default).

User Agent Switcher is useful when you’re deviating from the mainstream. Running Ubuntu pretty much flags you as a trouble maker or at least a dissident. Firefox maybe a bit less so, but you are indicating to advertisers that you don’t respect the expertise of those people far smarter than you who pre-installed IE (or Safari) to make your life easier. Set your user agent to IE 8 because the nail that sticks up gets pounded down.

Torbutton needs Tor to work. Tor provides really good privacy, but is a bit involved. The Tor Button Plugin for firefox makes it seem easier than it really is: you install it and click “use tor” and it looks like it is working but the first site you visit you get an proxy error because Tor isn’t actually running (DOH!).

To get Tor to work, you will have to open a terminal and do some command line fu before it will actually let you browse. Tor is also easier to install on Ubuntu than on Windows (at least for me, but as my browser history indicates I’m a bit of a miscreant dissident, so your mileage may vary).

Starting with these fine instructions.

sudu gedit /etc/apt/sources.list
add
deb http://deb.torproject.org/torproject.org/ lucid main
deb-src http://deb.torproject.org/torproject.org/ lucid main

Then run
gpg --keyserver keys.gnupg.net --recv 886DDD89
gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add -
sudo apt-get update
sudo apt-get upgrade
sudo apt-get update
sudo apt-get install tor tor-geoipdb

Install vidalia with the graphical ubuntu software center or with
sudo apt-get install vidalia

Tor expects Polipo. And vidalia makes launching and checking on Tor easier, so remove the startup scripts. (If Tor is running and you try to start it from vidalia, you get an uninformative error, vidalia has a “launch at startup” option, so let it run things.) Vidalia appears under the Applications->Network.

sudo update-rc.d -f tor remove

Polipo was installed with Tor, so configure it:
sudo gedit /etc/polipo/config

Clear the file (ctrl-a, delete)
paste in the contents of this file:

UPDATE: paste in the contents of this file:

(if the link above fails, search for “polipo.conf” to find the latest version)

I added the binary for polipo in Vidalia’s control panel, but that may be redundant (it lives in /usr/bin/polipo).

I had to reboot to get everything started.

And for private chats, consider OTR!

Posted at 17:45:45 GMT-0700

Category: PoliticsTechnology

Updating an IBM 366

Friday, May 15, 2009 

Today I’m updating an IBM 366-8863 to be a new home server because not having a quad 64 bit Xeon box with 24G of RAM and 6 x 72G SAS RAID 10 in your house would be like watching TV on a black and white CRT or something… and it was $350 on eBay so who could resist?  It will replace the old 5500 M20 and save 3U in the rack and probably a lot of power for a decent NAS box.

IBM eSeries x336

Unlike the 335, the 366 does not have a floppy drive.  There’s actually room in the cas right behind the IBM logo, next to the lightpath diagnostics and above the optical drive…  maybe I should get out my dremel and start looking for a 266Mhz 64bit PCI-X floppy controller.

The 366 is supported by the IBM Bootable Media Creator, which is a new thing for me.  This tool gathers all of the most recent firmware updates for the servers you specify (or all supported ones) and creates a single bootable disk (the 335 is not supported).   The tool found 23 updates for the 8863, though the versions are not all the same as you get doing the one-by-one download (there’s an option to select manually, but the integrated one-click approach is much easier).

All you do is download the creator tool for the OS of your choice, execute it, specify the systems you want to support, let it gather the updates and build the disk and it will even burn the disk for you.  Once the update disk is burned, you simply boot with it into a GUI (which supports normal mouse keyboard) and a few restarts later you have a fully patched machine.

The only thing left is to use the latest ServeRAID disk to update your ServeRAID configuration.

Nice job IBM!  This sort of thing is why I like IBM machines.  Plus they’re black.  And they have the built in  KVM/console controller over IP (remote supervisor II).

Posted at 18:45:49 GMT-0700

Category: FreeBSDTechnology

Updating an IBM 335

Sunday, May 10, 2009 

I’m bringing up an old IBM 335 for use as a pfSense Firewall.  It is a fine computer, with almost everything you’d want except dual power supplies (the 336 has those plus 64 bit hardware).

IBM 335 Server

The first step is updating the machine:

  • BIOS to 1.16: download the flash image, it writes itself to a floppy, boot with that floppy and flash the BIOS.  I had to go through a bunch of 1990’s era software disks until I found a few floppies that would format without errors.  This also updates the LSI 1030 disk controller.
  • Internal Diagnostics to 1.07: these are disk images (.img) diskcopy didn’t seem to do the right thing on my XP box, so I used diskwriter 0.9 to create the disks.  You boot off the BIOS update disk then select update diagnostics.
  • Configure the disks with ServeRAID.   I didn’t flash the BIOS on the controller, but I did reformat the disks and set them up as RAID 1.
  • Update the System Management Processor to 1.06.  This is a self-booting floppy.
  • Update the Broadcom NetXtreme NICs to 209h.  This is a self-booting floppy that creates a RAM disk then runs the update.  The command for the 335 is UPDATE 8830

This gets the core hardware up to date.  You might also want to flash the firmware in the disks, though I did not as my box is loaded with unsupported disks.  Plus 36GB SCSI disks aren’t exactly going through a lot of teething pains these days.

Then I installed pfSense from the LiveCD (verify the hash).  This is pretty effortless.  The only important bit of data is to set up the NICs: in the 335 under FreeBSD bge0 is the lower port and bge1 is the upper port.

At a later date I will install a 73P9265 Remote Supervisor II adapater, but the cable I have (73P9312) is for newer boxes.  The 335 needs the 02R1661: oddly it is cheaper to buy the cable with a card than just the cable.  This will probably need flashing of the firmware, but is a nice tool with remote KVM and a lot of other slick features.

Posted at 23:09:31 GMT-0700

Category: FreeBSDTechnology

Some good news in the world

Friday, April 4, 2008 

UAL 747

First, UAL has done two things that are quite good:

  • The ORD RCC’s former smoking room is now the “quiet room” with big barred out cell phone logos. It is quiet, and very pleasant.
  • T-Mobile is FREE at the RCC. FREE! Finally. I’ve been hassling UAL through 1kvoice etc. and the RCC comment cards since about 2001 – long before they even had T-Mobile – to provide free 802.11 to their members. Other clubs do. They got into this big provisioning contract with T-Mobile (or something) and years of comments went no where. It has been about 5 years, maybe the contract is up, but for whatever reason finally there is free WiFi at the RCC. Yay!

 

TrueCrypt

Truecrypt 5.1a supports sleep mode! YAY! 5.0 did not, it would crash entering sleep mode. It’s a cool thing, but crashing isn’t. I like the idea that if my laptop is stolen, my info is very likely secure, but not so much so that I can live without sleep mode, even risking a freezy RAM-cicle recovery. I sleep it getting on and off planes all the time and now I can. I’m very happy with this release.

Truecrypt finally died or got shut down or something, but VeraCrypt took the code and ran with it.

VeraCrypt Logo

Posted at 17:47:43 GMT-0700

Category: PlanesTechnologyTravel