technology

On technology

CoreELEC/Kodi on a Cheap S905x Box

Friday, September 11, 2020 

I got a cheap Android TV box off Amazon a while back and consolidated and it ended up being spare.  One of the problems with these TV boxes is they don’t get OS updates after release and soon video apps stop working because of annoying DRM mods.  I figured I’d try to switch one to Linux and see how it went.  There are some complications in that these are ARM devices (not x86/x64) and getting ADB to connect is slightly complicated due to it not being a USB device.  There are a few variations of the “ELEC” (Embedded Linux Entertainment Center), OpenELEC, LibreELEC, and CoreELEC (at least).  CoreELEC focuses on ARM boxes and makes sense for the cheapest boxes.

What you get is a little linux-based device that boots to Kodi, runs an SMB server, and provides SSH access, basically an open source SmartTV that doesn’t spy on you.  It is well suited to a media library you own, rather than rent, such as your DVD/Blu-Ray collection.  Perhaps physical media will come back into vogue as the rent-to-watch streaming world fractures into a million individual subscriptions.  Once moved to the FOSS world, there are regular updates and, generally, people committed to keeping things working and far better and far longer term support than OEM’s offer for random cheap android devices (or, even quite expensive ones).

A: Download the CoreELEC image:

You need to know the processor and RAM of your device – note that a lot of different brands are the same physical boxes.  My “CoCoq M9C Pro” (not listed) is also sold as a “Bqeel M9C Pro” (listed) and the SOC (S905X) and RAM (1G) match, so the Bqeel image worked fine.  The download helper at the bottom of the https://coreelec.org/ site got me the right file easily.

B: Image a uSD or USB device for your wee box:

I’m running Linux as my desktop so I used the LibreELEC Creator Tool and just selected the file downloaded previously.  That part went easily.  The only issue was that after imaging, Linux didn’t recognize the device and I had to systemctl --user restart gvfs-udisks2-volume-monitor before I could do the next step.

C: One manual file move for bootability:

ELEC systems use a “device tree” – a file with hardware descriptors that are device specific to get (most) of the random cheap peripherals working on these cheap boxes.  You have to find the right one in the /Device Trees folder on your newly formatted uSD/USB device and copy it to the root directory and rename it dtb.img.

D: Awesome, now what?

Once you have your configured formatted bootable uSD/USB you have to get your box to boot off it.  This is a bit harder without power/volume control buttons normally on Android devices.  I have ADB set up to strip bloatware from phones.  You just need one command to get this to work, reboot update, but to get it to your box is a bit of a trick.

Most of these boxes are supplied rooted, so you don’t need to do that, but you do need a command line tool and to enable ADB.

To enable ADB: navigate to the build number (usually Settings > System > About phone > Build number) and click it a bunch of times until it says “you’re a developer!”  Then go to developer options and enable ADB.  You can’t use it yet though because there’s no USB.

Install a Terminal Emulator for Android and enable ADB over TCP

su
setprop service.adb.tcp.port 5555
stop adbd
start adbd

Then check your IP with ifconfig and on your desktop computer run

adb connect dev.ip.add.ress:5555

(dev.ip.add.ress is the IP address of your device – like 192.168.100.46, but yours will be different)

Then execute adb reboot update from the desktop computer and the box should reboot then come up in CoreELEC.

Going back to android is as easy as disconnecting the boot media and rebooting, the android OS is still on the device.

Issue: WIFI support.

Many of these cheap devices use the cheapest WIFI chips they can and often the MFGs won’t share details with FOSS developers, so driver support is weak.  Bummer, but the boxes have wired connections and wired works infinitely better anyway: it isn’t a phone, it’s attached to a TV that’s not moving around, run the damn wire and get a stable, reliable connection.  If that’s not possible check the WIFI chips before buying or get a decent USB-WIFI adapter that is supported.

 

Posted at 16:46:48 GMT-0700

Category: HowToLinuxtechnology

WebP and SVG

Tuesday, September 1, 2020 

Using WebP coded images inside SVG containers works.  I haven’t found any automatic way to do it, but it is easy enough manually and results in very efficiently coded images that work well on the internets.  The manual process is to Base64 encode the WebP image and then open the .svg file in a text editor and replace the

xlink:href="data:image/png;base64, ..."

with

xlink:href="data:image/webp;base64,..."

(“…” means the appropriate data, obviously).


Back in about 2010 Google released the spec for WebP, an image compression format that provides a roughly 2-4x coding efficiency over the venerable JPEG (vintage 1974), derived from the VP8 CODEC they bought from ON2. VP8 is a contemporary of and technical equivalent to H.264 and was developed during a rush of innovation to replace the aging MPEG-II standard that included Theora and Dirac. Both VP8 and H.264 CODECs are encumbered by patents, but Google granted an irrevocable license to all patents, making it “open,” while H.264s patents compel licensing from MPEG-LA.  One would think this would tend to make VP8 (and the WEBM container) a global standard, but Apple refused to give Google the win and there’s still no native support in Apple products.

A small aside on video and still coding techniques.

All modern “lossy” (throwing some data away like .mp3, as opposed to “lossless” meaning the original can be reconstructed exactly, as in .flac) CODECs are founded on either Discrete Cosine Transform (DCT) or Wavelet (DWT) encoding of “blocks” of image data.  There are far more detailed write ups online that explain the process in detail, but the basic concept is to divide an image into small tiles of data then apply a mathematical function that converts that data into a form which sorts the information from least human-perceptible to most human-perceptible and sets some threshold for throwing away the least important data while leaving the bits that are most important to human perception.

Wavelets are promising, but never really took off, as in JPEG2000 and Dirac (which was developed by the BBC).  It is a fairly safe bet that any video or still image you see is DCT coded thanks to Nasir Ahmed, T. Natarajan and K. R. Rao.  The differences between 1993’s MPEG-1 and 2013’s H.265 are largely around how the data that is perceptually important is encoded in each still (intra-frame coding) and some very important innovations in inter-frame coding that aren’t relevant to still images.

It is the application of these clever intra-frame perceptual data compression techniques that is most relevant to the coding efficiency difference between JPEG and WebP.

Back to the good stuff…

Way back in 2010 Google experimented with the VP8 intra-coding techniques to develop WebP, a still image CODEC that had to have two core features:

  • better coding efficiency than JPEG,
  • ability to handle transparency like .png or .tiff.

This could be the one standard image coding technique to rule them all – from icons to gigapixel images, all the necessary features and many times better coding efficiency than the rest.  Who wouldn’t love that?

Apple.

Of course it was Apple.  Can’t let Google have the win.  But, finally, with Safari 14 (June 22, 2020 – a decade late!) iOS users can finally see WebP images and websites don’t need crazy auto-detect 1974 tech substitution tricks.  Good job Apple!

It may not be a coincidence that Apple has just released their own still image format based on the intra-frame coding magic of H.265, .heif and maybe they thought it might be a good idea to suddenly pretend to be an open player rather than a walled-garden-screw-you lest iOS insta-users wall themselves off from the 90% of the world that isn’t willing to pay double to pose with a fashionable icon in their hands.  Not surprisingly, .heic, based on H.265 developments is meaningfully more efficient than WebP based on VP8/H.264 era techniques, but as it took WebP 10 years to become a usable web standard, I wouldn’t count on .heic  having universal support soon.  I expect Google is hard at work on a VP9-based version of WebP right now.  Technology is fast, marketing is slow.  Oh well.  In the mean time….

SVG support in browsers is a much older thing – Apple embraced it early (SVG was not developed by Google so….) and basically everything but IE has full support (IE…  the tool you use to download a real browser).  So if we have SVG and WebP, why not both together?

Oddly I can’t find support for this in any of the tools I use, but as noted at the open, it is pretty easy.  The workflow I use is to:

  • generate a graphic in GIMP or Photoshop or whatever and save as .png or .jpg as appropriate to the image content with little compression (high image quality)
  • Combine that with graphics in Inkscape.
  • If the graphics include type, convert the type to SVG paths to avoid font availability problems or having to download a font file before rendering the text or having it render randomly.
  • Save the file (as .svg, the native format of Inkscape)
  • Convert the image file to WebP with a reasonable tool like Nomacs or Ifranview.
  • Base64 encode the image file, either with base64 # base64 infile.webp > outfile.webp.b64 or with this convenient site
  • If you use the command line option the prefix to the data is “data:image/webp;base64,
  • Replace the … on the appropriate xlink:href="...." with the new data using a text editor like Atom.
  • Drop the file on a browser page to see if it works.

WordPress blocks .svg uploads without a plugin, so you need one

The picture is 101.9kB and tolerates zoom quite well. (give it a try, click and zoom on the image).

 

Posted at 08:54:16 GMT-0700

Category: HowToLinuxphotoself-publishingtechnology

Dealing with Apple Branded HEIF .HEIC files on Linux

Saturday, August 22, 2020 

Some of the coding tricks in H.265 have been incorporated into MPEG-H coding, an ISO standard introduced in 2017, which yields a roughly 2:1 coding efficiency gain over the venerable JPEG, which was introduced in 1992.  Remember that?  I do; I’m old.  I remember having a hardware NUBUS JPEG decoder card.   One of the reasons JPEG has lasted so long is that images have become a small storage burden (compared to 4k video, say) and that changing format standards is extremely annoying to everyone.

Apple has elected to make every rational person’s life difficult and put a little barbed wire around their high-fashion walled garden and do something a little special with their brand of a HEVC (h.265) profile for images.  Now normally seeing iOS user’s insta images of how fashionable they are isn’t really worth the effort, but now and then a useful correspondent joins the cult and forks over a ton of money to show off a logo and starts sending you stuff in their special proprietary format.  Annoying, but fixable.

Assuming you’re using an OS that is neither primarily spyware nor fashion forward, such as Linux Mint, you can install HEIF decode (including Apple Brand HEIC) with a few simple commands:

$ sudo add-apt-repository ppa:jakar/qt-heif
$ sudo apt update
$ sudo apt install qt-heif-image-plugin

Once installed, various image viewers should be able to decode the images.  I rather like nomacs as a fairly tolerable replacement for Irfan Skiljan‘s still awesome irfanview.

Posted at 03:56:36 GMT-0700

Category: HowToLinuxphotoPositivereviewstechnology

Integrate Fail2Ban with pfSense

Monday, July 13, 2020 

Fail2Ban is a very nice little log monitoring tool that is used to detect cracking attempts on servers and to extract the malicious IPs and do the things to them – usually temporarily adding the IP address of the source of badness to the server’s firewall “drop” list so that IP’s bad packets are lost in the aether.   This is great, but it’d be cool to, instead of running a firewall on every server each locally detecting and blocking malicious actors, to instead detect across all services and servers on the LAN and push the results up to a central firewall so the bad IPs can’t reach the network at all.

I like pfSense as a firewall and run FreeBSD on my servers; I couldn’t find a prebuilt tool to integrate F2B with pfSense, but it wasn’t hard to hack something together so it worked. Basically I have F2B maintain a local “block list” of bad IPs as a simple text file which is published via Apache from where pfSense’s grabs it and applies it as a LAN-wide IP filter.  I use the pfSense package pfBlockerNG to set up the tables but in the end a custom script running on the pfSense server actually grabs the file and updates the pfSense block lists from it on a 1 minute cron job.

There are plenty of well-written guides for getting F2B working and how to configure it for jails; I found the following useful:

The custom bits I did to get it to work are:

Custom F2B Action

On the protected side, I modified the “dummy.conf” script to maintain a list of bad IPs in an Apache served location that pfSense could reach.  F2B manages that list, putting bad IPs in “jail” and letting them out as in any normal F2B installation – but instead of being the local server’s packet filter, it is a web-published text list.

# Fail2Ban configuration file
#
# Author: David Gessel
# Based on: dummy.conf by Cyril Jaquier
#

[Definition]

# Option:  actionstart
# Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
# Values:  CMD
#


actionstart = if [ -z '' ]; then 
                  touch 
                  printf %%b "# \n" 
                  fi 
              chmod 755 
              echo "%(debug)s started"

# Option:  actionflush
# Notes.:  command executed once to flush (clear) all IPS, by shutdown (resp. by stop of the jail or this action)
# Values:  CMD
#

actionflush = if [ ! -z '' ]; then
                  rm -f 
                  touch 
                  printf %%b "# \n" 
                  fi
              chmod 755 
              echo "%(debug)s clear all"

# Option:  actionstop
# Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
# Values:  CMD
#
actionstop = if [ ! -z '' ]; then
                  rm -f 
                  touch 
                  printf %%b "# \n" 
                  fi
             chmod 755 
             echo "%(debug)s stopped"

# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
actioncheck =

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#

actionban = printf %%b "\n" 
            sed -i '' '/^$/d' 
            sort -u  -o 
            chmod 755 
            echo "%(debug)s banned  (family: )"

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#

# flush the IP using grep which is supposed to be about 15x faster than sed  
# grep -v "pattern" filename > filename2; mv filename2 filename


actionunban = grep -v ""  > 
              mv  
              chmod 755 
              echo "%(debug)s unbanned  (family: )"


debug = []   --

[Init]

init = BRT-DNSBL

target = /usr/jails/claudel/usr/local/www/data-dist/brt/dnsbl/brtdnsbl.txt
temp = .tmp
to_target = >> 

Once this list is working, then move to the pfSense side.

Set up pfBlockerNG

The basic setup is well described, for example in https://protectli.com/kb/how-to-setup-pfblockerng/ and it provides a lot of useful blocking options, particularly with externally maintained lists of internationally recognized bad actors.  There are two basic functions, related but different:

DNSBL

Domain Name Service Block Lists are lists of domains associated with unwanted activity and blocking them at the DNS server level (via Unbound) makes it hard for application level services to reach them.  A great use of DNSBLs is to block all of Microsoft’s telemetry sites, which makes it much harder for Microsoft to steal all your files and data (which they do by default on every “free” Windows 10 install, including actually copying your personal files to their servers without telling you!  Seriously.  That’s pretty much the definition of spyware.)

It also works for non-corporate-sponsored spyware, for example lists of command and control servers found for botnets or ransomware servers.  This can help prevent such attacks by denying trojans and viruses access to their instruction servers.  It can also easily help identify infected computers on the LAN as any blocked requests are logged (to 1.1.1.1 at the moment, which is an unfortunate choice given that is now a well-reputed DNS server like Google’s 8.8.8.8 but, it seems, without all the corporate spying.)  There is a bit of irony in blocking lists of telemetry gathering IPs that are built using telemetry.

Basically DNSBLs prevent services on the LAN from reaching nasty destinations on the internet by returning any DNS request to convert a malicious domain name to a dead-end IP address.  When your windows machine wants to report your web browsing habits to microsoft, it instead gets a “page not found” error.

IPBL

This concept uses an IPBL, a list of IP addresses to block.  An IPBL works at a lower level than a DNSBL and typically is set up to block traffic in both directions – a script kiddie trying to brute force a password can be blocked from reach the services on the LAN, but so too can the reverse direction be blocked – if a malicious entity trips F2B, not only are they stopped, so too are any sneaky services on your LAN blocked from reaching them on the internet.

All we need to do is get the block list F2B is maintaining into pfSense.  pfBlockerNG can subscribe to the list easily enough, but the minimum update time is an hour – which is an awfully long time to let someone try to guess passwords or flood your servers with 404 requests or whatever else you’re using F2B to detect and stop.  So I wrote a simple script that executes a few simple commands to grab the IP list F2B maintains, clean it, and use it to update the packet filter drop lists:

/root/custom/brtblock.sh

#!/usr/bin/env sh
# set -x # uncomment for "debug"

# Get latest block list
/usr/local/bin/curl -m 15 -s https://server.ip/brtdnsbl.txt > /var/db/pfblockerng/original/BRTDNSBL.orig
# filter for at least semi-valid IPs.
/usr/bin/grep  -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' /var/db/pfblockerng/original/BRTDNSBL.orig > /var/db/pfblockerng/native/BRTDNSBL.txt
# update pf tables
/sbin/pfctl -t pfB_BRTblock -T replace -f /var/db/pfblockerng/native/BRTDNSBL.txt > /dev/null 2>&1

HT to Jared Davenport for helping to debug the weird /env issues that arise when trying to call these commands directly from cron.

Preventing Self-Lockouts

One of the behaviors of pfBlockerNG that the dev seems to think is a feature is automatic filter order management.  This overrides manually sorted filter orders and puts pfB’s block filters ahead of all other filters, including, say, allow filters of your own IPs that you don’t want to ever be locked out in case you forget your passwords and accidentally trigger F2B on yourself.  To fix this, you have to use a non-default setting and make all IP block list “action” types “Alias_Native.”

pfBlockerNG Native IP Block Lists

Then you write your own per-alias filter (typically “drop” or “reject”) and pfBlockerNG won’t auto-order them for you on update.

pfSense Filter Order

Cron Plugin

The last ingredient is to update the list on pfSense quickly.  pfSense is designed to be pretty easy to maintain, so it overwrites most of the file structure on update, making command line modifications frustratingly transient.  I understand that /root isn’t flushed on an update so the above script should persist inside the /root directory.  But crontab -e just doesn’t stick around.  To have cron modifications persist, install the “Cron” package with the pfSense package manager.  Then just set up a cron job to run the script above to keep the block list updated.  “*/1” means run the script once a minute.

pfSense Cron Config

Results

The system seems to be working well enough – the list of miscreants as small, but effectively targeted, 11,840 packets dropped from an average of about 8-10 bad IPs at any given time.

pfBlockerNG current status

Posted at 05:48:43 GMT-0700

Category: FreeBSDHowToSecuritytechnology

Save your email! Avoid the Thunderbird 68 update

Thursday, November 28, 2019 

TLDR:

If you’ve customized TB with plugins you care about, DO NOT UPDATE to 68 until you verify that every plugin you use is compatible.  TB will NOT check for you and once you launch 68, the plugins that have been updated to 68 compatibility will not work with 60.x, which means you better have a backup of your .thunderbird profile folder or you’re going to be filled with seething rage and you’ll have to undo the update. This misery is the consequence of Mozilla having failed to fully uphold their obligation to the user and developer communities that rely on and have enhanced the tools they control.

BTW: if you’re using Firefox and miss the plugins that made it more than just a crappy clone of Chrome, Waterfox is great and actually respects users and community  developers.  Give it a try.

Avoid Thunderbird 68 Hell

To avoid this problem now and in the future, you have to disable automatic updates.  In Thunderbird: Edit->Preferences->Advanced->General-[Config Editor…]->app.update.auto=False, app.update.enabled=False.

Screenshot of no update prefs thunderbird

On Linux, you should also disable OS Updates using Synaptic: select installed thunderbird 60.x and then from the menu bar Package->Lock Version.

screenshot of locking package in synaptic

If you’ve been surprise updated to the catastrophically incompatible developer vanity project and massive middle finger to the plugin developer community which is 68 (and 60 to a lesser extent), then you have to revert.  This sucks as 60.x isn’t in the repos.

Undo Thunderbird 68 Hell

First, do not run 68.  Ever.  Don’t.  It will cause absolute chaos with your plugins.  First it showed most incompatible, then updated some, then showed others compatible, but had deleted the .xpi files so they weren’t in the .thunderbird folder any more, despite being listed and shown incorrectly as compatible.  This broke some things I could live without like Extra Format Buttons, but others I really needed like Dorando Keyconfig and Sieve.  Mozilla’s attitude appears to be “if you’re using software differently than we think you should, you’re doing it wrong.”

The first step before breaking things even more is to backup your .thunderbird directory.  You can find the location from Help->Troubleshooting Information->Application Basics->Profile Directory.  Just click [Open Directory].  Make a backup copy of this directory before doing anything else if you don’t already have one, in linux a command might be:

tar -jcvf thunderbird_profile_backup.tar.bz2 .thunderbird

If you’re running Windows, old installers of TB are available here.

In Linux, using a terminal, see what versions are available in your distro:

apt-cache show thunderbird

I see only 1:68.2.1+build1-0ubuntu0.18.04.1 and 1:52.7.0+build1-0ubuntu1. Oh well, neither is what I want. While in the terminal uninstall Thunderbird 68

sudo apt-get remove thunderbird

As my distro, Mint 19.2, only has 68.x and 52.x in the apt cache, I searched to find a .deb file of a recent version.  I couldn’t find the last plugin compatible version, 60.9.0 as an easy to install .deb (though it is available for manual install from Ubuntu) so I am running 60.8.0, which works.  One could download the executable file of 60.9.1 .and put it somewhere (/opt, say) and then update start scripts to execute that location.

I found the .deb file of 60.8.0 at this helpful historical repository of Mozilla installers.  Generally the GUI will auto-install on execution of the download.  But don’t launch it until you restore your pre-68 .thunderbird profile directory or it will autocreate profile files that are a huge annoyance.  If you don’t have a pre-68 profile, you will probably have to hunt down pre-68 compatible versions of all of your plugins, though I didn’t note any catastrophic profile incompatibilities (YMMV).

Good luck. Mozilla just stole a day of your life.

Read more…

Posted at 07:51:32 GMT-0700

Category: HowTotechnology

Frequency of occurrence analysis in LibreOffice

Tuesday, November 19, 2019 

One fairly common analytic technique is finding out, for example, the rate at which something appears in a time referenced file, for example a log file.

Lets say you’re looking for the reate of some reported failure to determine, say, whether a modification or update had made it better or worse.  There are log analysis tools to do this (like Splunk), but one way to do it is with a spreadsheet.

Assuming you have a table with time in a column (say A) and some event text in another (say B) like:

11-19 16:51:03 a bad error happened
11-19 16:51:01 something minor happened
11-19 12:51:01 cat ran by

you might convert that event text to a numerical value (into, say, column C) for example by:

=IF(ISERROR(FIND("error",B1)),"",1)

FIND returns true if the text (“error”) is found, but returns an error if not. ISERROR inverts that and returns logical values for both. The IF ISERROR construction allows one to specify values if the text is found or not – a bit complex but the result in this case will be “” (blank) if “error” isn’t found in B1 and 1 if “error” is found.

Great, fill down and you have a new column C with blank or 1 depending if “error” was found in column B. Summing column C yields the total number of lines in which the substring “error” occured.

But now we might want to make a histogram with a sum count of the occurrences within a specific time period, say “errors/hour”.

Create a new column, say column D, and fill the first two rows with date/time values separated by the sampling period (say an hour), for example:

11/19/2019 17:00:00
11/19/2019 16:00:00

And fill down; there’s a quirk where LibreOffice occasionally loses one second, which looks bad. It probably won’t meaningfully change the results, but just edit the first error, then continue filling down.

To sum within the sample period use COUNTIFS in (say) column E to count the occurrences in entire columns that meet a string of criterion: in this case three criterion have to be met: the value of C is 1 (not “”), The value of A (time) is before the start of the sampling period (D1) and after the end (D2). That is:

=COUNTIFS(C1:C500,"1",$A1:$A500,">="&$D2,$A1:$A500,"<"&$D1)

Filling this formula down populates the sampling periods with the count per sampling period, effectively the occurrence rate per period, in our example errors/hour.

Posted at 08:36:41 GMT-0700

Category: HowTotechnology

Lets encrypt with security/dehydrated (acme-client is dead)

Thursday, June 27, 2019 

Well….  security/acme-client is dead.  That’s sad.

Long live dehydrated, which uses the same basic authentication method and is pretty much a drop in replacement (unlike scripts which use DNS authentication, say).

In figuring out the transition, I relied on the following guides:

If you’re migrating from acme-client, you can delete it (if you haven’t already)

portmaster -e acme-client

And on to installation.  This guide is for libressl/apache24/bash/dehydrated.  It assumes you’ve been using acme-client and set it up more or less like this.

Installation of what’s needed

if you don’t have bash installed, you will. You can also build with ZSH but set the config before installing.

cd /usr/ports/security/dehydrated && make install clean && rehash

or

portmaster security/dehydrated

This guide also uses sudo, if it isn’t installed:

cd /usr/ports/security/sudo && make install clean && rehash

or

portmaster /security/sudo

Set up directories and accounts

mkdir -p /var/dehydrated
pw groupadd -n _letsencrypt -g 443
pw useradd -n _letsencrypt -u 443 -g 443 -d /var/dehydrated -w no -s /nonexistent
chown -R _letsencrypt /var/dehydrated

If migrating from acme-client this should be done but:

mkdir -p -m 775 /usr/local/www/.well-known/acme-challenge
chgrp _letsencrypt /usr/local/www/.well-known/acme-challenge

# If migrating from acme-client

chmod 775 /usr/local/www/.well-known/acme-challenge
chown -R _letsencrypt /usr/local/www/.well-known

Configure Dehydrated

ee /usr/local/etc/dehydrated/config

add/adjust

014 DEHYDRATED_USER=_letsencrypt

017 DEHYDRATED_GROUP=_letsencrypt

044 BASEDIR=/var/dehydrated

056 WELLKNOWN="/usr/local/www/.well-known/acme-challenge"

065 OPENSSL="/usr/local/bin/openssl"

098 CONTACT_EMAIL=gessel@blackrosetech.com

save and it should run:

su -m _letsencrypt -c 'dehydrated -v'

You should get roughly the following output:

# INFO: Using main config file /usr/local/etc/dehydrated/config
Dehydrated by Lukas Schauer
https://dehydrated.io

Dehydrated version: 0.6.2
GIT-Revision: unknown

OS: FreeBSD 11.2-RELEASE-p6
Used software:
bash: 5.0.7(0)-release
curl: curl 7.65.1
awk, sed, mktemp: FreeBSD base system versions
grep: grep (GNU grep) 2.5.1-FreeBSD
diff: diff (GNU diffutils) 2.8.7
openssl: LibreSSL 2.9.2

File adjustments and scripts

by default it will read /var/dehydrated/domains.txt for the list of domains to renew

Migrating from acme-client? Reuse your domains.txt, the format is the same.

mv /usr/local/etc/acme/domains.txt /var/dehydrated/domains.txt

Create the deploy script:

ee /usr/local/etc/dehydrated/deploy.sh

The following seems to be sufficient

#!/bin/sh

/usr/local/sbin/apachectl graceful

and make executable

chmod +x /usr/local/etc/dehydrated/deploy.sh

Give the script a try:

/usr/local/etc/dehydrated/deploy.sh

This will test your apache config and that the script is properly set up.

There’s a bit of a pain in the butt in as much as the directory structure for the certs changed. My previous guide would put certs at /usr/local/etc/ssl/acme/domain.com/cert.pem, this puts them at /var/dehydrated/certs/domain.com

Check the format of your certificate references and use/adjust as needed. This worked for me – note you can set your key locations to be the same in the config file, but the private key directory structure does change between acme-client and dehydrated.

sed -i '' "s|/usr/local/etc/ssl/acme/|/var/dehydrated/certs/|" /usr/local/etc/apache24/extra/httpd-vhosts.conf

Or if using httpd-ssl.conf

sed -i '' "s|/usr/local/etc/ssl/acme/|/var/dehydrated/certs/|" /usr/local/etc/apache24/extra/httpd-ssl.conf

And privkey moves from /usr/local/etc/ssl/acme/private/domain.com/privkey.pem to /var/dehydrated/certs/domain.com/privkey.pem so….

sed -i '' "s|/var/dehydrated/certs/private/|/var/dehydrated/certs/|" /usr/local/etc/apache24/extra/httpd-vhosts.conf

# or

sed -i '' "s|/var/dehydrated/certs/private/|/var/dehydrated/certs/|" /usr/local/etc/apache24/extra/httpd-ssl.conf

Git sum certs

su -m _letsencrypt -c 'dehydrated --register --accept-terms'

Then get some certs

su -m _letsencrypt -c 'dehydrated -c'

-c is “chron” mode which is how it will be called by periodic.

and “deploy”

/usr/local/etc/dehydrated/deploy.sh

If you get any errors here, track them down.

Verify your new certs are working

cd /var/dehydrated/certs/domain.com/
openssl x509 -noout -in fullchain.pem -fingerprint -sha256

Load the page in the browser of your choice and view the certificate, which should show the SHA 256 fingerprint matching what you got above.  YAY.

Automate Updates

ee /etc/periodic.conf

insert the following

weekly_dehydrated_enable="YES"
weekly_dehydrated_user="_letsencrypt"
weekly_dehydrated_deployscript="/usr/local/etc/dehydrated/deploy.sh"
weekly_dehydrated_flags="-g"

note the flag is –keep-going (-g) Keep going after encountering an error while creating/renewing multiple certificates in cron mode

Posted at 11:38:45 GMT-0700

Category: FreeBSDSecuritytechnology

Getting the postfixadmin 3.2 update to work with FreeBSD

Monday, September 17, 2018 

Postfixadmin is a very nice tool for managing a mail server via a nice web interface that just went through an update to add some security and compatibility features, but at the current revision there are a few bugs (the maintainer says these will all be cleared up in the next release).  A few work-arounds:

If you get:

pkg-static: Unable to access file /var/ports/usr/ports/mail/postfixadmin/work/stage/usr/local/share/postfixadmin/README.md:No such file or directory 

Then run

 # make config

and enable DOCS.

If you get

PHP Fatal error:  Uncaught exception 'PharException' with message 'phar "/usr/local/www/postfixadmin/lib/random_compat.phar" openssl signature could not be verified: openssl not loaded' in /usr/local/www/postfixadmin/lib/random_compat.phar:8\nStack trace:\n#0 /usr/local/www/postfixadmin/lib/random_compat.phar(8): Phar::webPhar(NULL, 'index.php')\n#1 /usr/local/www/postfixadmin/common.php(72): require_once('/usr/local/www/...')\n#2 /usr/local/www/postfixadmin/public/common.php(2): require_once('/usr/local/www/...')\n#3 /usr/local/www/postfixadmin/public/setup.php(27): require_once('/usr/local/www/...')\n#4 {main}\n  thrown in /usr/local/www/postfixadmin/lib/random_compat.phar on line 8

There’s a dependency that’s not built into the makefile yet.

Run

 # portmaster security/php56-openssl

(adjust the PHP version to match, or the install command to suit your environment).  Remember to run # apachectl restart.

Also note that the directory the code is served from has been moved to the subdirectory /public for security. This may require updating URLs, DocumentRoot, or modrewrite as appropriate to the webserver environment to get to the login page.

After updating, navigate to public/upgrade.php to update the database automatically.


And because this is open source and not some horrible closed source product, it took a whole 2 hours for a fix.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231424

Thanks ports.maintainer@evilphi.com!

Posted at 02:20:58 GMT-0700

Category: FreeBSD

How to stick with a decent version of Firefox (pre-Quantum)

Friday, June 1, 2018 

Firefox (52) remains my browser of choice – entirely because of plug-ins. When Firefox completely destroyed the UI/UX with “Australis,” a horrific UI change that basically made Firefox into a crappy clone of Chrome, the only thing that made Firefox usable was “Classic Theme Restorer.” Apparently, unsatisfied with the damage Mozilla had managed to wreak on their user-base with idiotic UI decisions, over the past year or so, a new version called “Quantum” (57) was rolled out that broke the functionality of almost every important plug-in.

This utterly disastrous and truly unforgivable transgression against the user-base was only slightly mitigated by sustaining 52-ESR, at least until the Sept of this year. After that, everyone who cares about having a decent alternative to Chrome will have to migrate to Waterfox.

In the mean time, one really important thing you have to remember to do if you stuck reinstalling your system on Linux (e.g. Linux Mint) is to immediately uninstall Firefox before using it even once.  Then change your install version to ESR and install. If you let Quantum run even once, it will mark all your good plugins as disabled and you need to reinstall them one at a time to get them working again.

sudo add-apt-repository ppa:jonathonf/firefox-esr
sudo apt-get update
sudo apt-get install firefox-esr

Well, that’s the end of Firefox….  Sad to see it go after all these years, but the new plugin concept has made Firefox a subordinate version of Chrome rather than a powerful, customizable tool.

 

I followed these fine directions and now have waterfox running.

echo 'deb https://dl.bintray.com/hawkeye116477/waterfox-deb release main' | sudo tee /etc/apt/sources.list.d/waterfox.list
curl https://bintray.com/user/downloadSubjectPublicKey?username=hawkeye116477 | sudo apt-key add -
sudo apt-get update
sudo apt upgrade
sudo apt install waterfox

 
Just import your firefox preferences on starup and walla, you instant happiness with all the plugins that Quantum broke restored, including such absolute essentials as “Classic Theme Restorer” (which undoes the absolutely horrible UI changes that Mozilla adopted) and downthemall, privacy plugins, etc. If you’ve updated some plugins to be Quantum compatible, you’ll have to back those up (for me that’s things like FoxClocks and Noscript, which managed to hack together semi-viable Quantum compatible plugins after slogging through Mozilla’s buggy WebExtensions API and HTML5 quirks).

This whole translation is quite unfortunate. Waterfox is dependent on the Firefox code base, so this solution may have a finite lifespan, but for now it works and undoes the horror of Firefox 57+/Quantum.

Posted at 11:43:58 GMT-0700

Category: Neutralreviewstechnology

Rubygem passenger flavors in FreeBSD

Saturday, February 3, 2018 

The latest bit of code to adopt the trendy new “flavors” model is passenger.  As “flavors” aren’t supported in Portmaster, this means the transition results in errors  like:

/bin/rmdir /var/ports/usr/ports/www/rubygem-passenger/work-apache/stage/usr/local/lib/ruby/gems/2.4/extensions 2> /dev/null || true
( cd /var/ports/usr/ports/www/rubygem-passenger/work-apache/passenger-5.1.12 &&  /bin/sh -c '(/usr/bin/find -Ed $1 $3 | /usr/bin/cpio -dumpl $2 >/dev/null 2>&1) &&  /usr/bin/find -Ed $1 $3 \(   -type d -exec /bin/sh -c '\''cd '\''$2'\'' && chmod 755 "$@"'\'' . {} +  -o -type f -exec /bin/sh -c '\''cd '\''$2'\'' && chmod 0644 "$@"'\'' . {} + \)' COPYTREE_SHARE buildout /var/ports/usr/ports/www/rubygem-passenger/work-apache/stage/usr/local/lib/ruby/gems/2.4/gems/passenger-5.1.12 )
find: buildout: No such file or directory
find: buildout: No such file or directory

You can make with flavors manually or (assuming your current version is 5.1.12, adjust as needed):

# portmaster -o www/rubygem-passenger rubygem-passenger-5.1.12
Posted at 04:58:54 GMT-0700

Category: FreeBSDtechnology