Let’s Encrypt….

Sunday, December 10, 2017 

Let’s encrypt, why not?

Wanna know how I did it for FreeBSD/Apache/acme-client, jump below.

Let’s encrypt is a service from the fine people at Mozilla, who, when they’re not trying to prove that Firefox can be a Chrome clone, do some really good stuff. Certificates are what give you the little warm fuzzy feeling of a green lock icon, and when properly configured, avoid giving you that terrifying feeling that something horrible is about to happen if you visit a site with an expired one or a self-signed one.

There are some huge structural problems in the certificate concept that seem to exist only to validate the certificate mafia, that can charge $100s per year for a validated certificate, as if executing the script to issue one was somehow expensive. It is not, you can generate one yourself that provides exactly the same security as one provided by a big company that gets their root certs distributed in a browser, but browsers reject these with scary messages so webmasters have to keep buying them.

Now there’s a theory behind why they’re ripping you off: the premise is that the certificate verifies the site is who it says it is – that if you go to, you’re actually visiting your real bank, not being redirected by a man-in-the-middle attack to some fake landing page to harvest your passwords, log into your account, and steal all your cats.  There are a few problems with this:

  • Nobody actually checks a URL so while a certificate sort of adds some weight to the probability that is owned by mybank, not some hacker a few tables over ARP poisoning the cafe wifi, it doesn’t do anything if you click on a link to
  • The companies that claim to check IDs and verify owners, do not.  That would cost money. You think they’re gonna actually do that?  No… (CAcert actually does, but they don’t get a root cert because… they do it for free. And don’t have Mozilla’s money and clout.)
  • Stealing a root cert private key can generate significant LOLZ; it happens a lot.
  • Law enforcement the world over has “lawful intercept” certs.  You’re probably on some country’s poop list if you have ever used social media. Their laws permit intercepting your communications.  Some country’s laws somewhere certainly do no matter who you are.
  • But dang, those annoying warnings that do nothing to secure you mean that people who publish a website just for the good of the planet either have to pay up, go through a lot of hassle, or leave their user’s content streams exposed to the world’s prying eyes…

…Until Let’s Encrypt came along.  It is a lovely little set of tools and services that not only issue browser-accepted certs (see the green lock?) but also automate renewal.  They basically check that you have enough control over your website to let a script write a file that that they can read back and verify, and if so, you’re who you say you are: the person with write access to the server powering the website they’re giving the certificate too.  That’s all anyone can really do, and is as secure as any other cert there is for identification of a site: that is except for stolen certs, url typos, law enforcement certs, or malicious code on your computer, if you visit and you don’t get any warnings, you’re probably reading data coming off my computer and not some hacker pretending to be me.

I got Let’s Encrypt to work, but it took some modifications of the existing guides, and I think the service is a good thing that more people should use, so in the spirit of investing some of my resources into the great shared experiment that is Open Source, here’s my How To:

Upstream Guides:

I found these two guides extremely helpful.

Step 1: Installing the certificate generation tool

There are a few different software tools to manage the Let’sEncrypt process.  I elected to use Kristaps Dzonsons acme-client, ported to FreeBSD by Bernard Spil.

I was using OpenSSL on my site.  Bernard and Kristaps have some strong opinions on OpenSSL and heartbleed and a few other problems and therefore require LibreSSL.   If you’re using it already, great.  If not, you’ll have to install it.  It wasn’t too terrible, but I ran into a few issues:
Or, easy peasy

# ee /etc/make.conf
set  DEFAULT_VERSIONS+= ssl=libressl
# portmaster -od security/libressl security/openssl
# portmaster -rd security/libressl

if that fails with

===>>> The argument to -r must be a package name, or a glob pattern

Then try:

# pkg version -v | grep libre
libressl-2.6.3 = up-to-date with index
# portmaster -rd libressl-2.6.3
or for a complete refresh
# portmaster -Rafd

Curl will probably fail with LibreSSL (and with the latest, if it has brotli support enabled).  Check the google to see if these fixes are still needed, or just:

# cd /usr/ports/ftp/curl
# make config

disable TLS-SRP

ftp/curl 7.75.0 has an issue with pied piper brotli, which requires modifying the makefile to build --without-brotli as indicated in comment #2 

(Sunpoet, the curl port maintainer, got back to me with an update: when PR/223966 is integrated in Brotli, he will add an optional Brotli support flag and it should work fine at that point without the Makefile edit.)

Step 2: Actually installing acme-client

The really easy part: you should be able to

# portmaster security/acme-client

and be on your way to configuration heaven.

Step 3: Initial configuration

The defaults for acme-client expect certain directories to exist and the installer doesn’t create them.

# mkdir -pm750 /usr/local/www/.well-known && chown -R www:www /usr/local/www/.well-known
# mkdir -pm750 /usr/local/www/.well-known/acme-challenge && chown -R www:www /usr/local/www/.well-known/acme-challenge

The how-to’s seemed to forget the last one.

And make a modification to your httpd.conf file to permit the Let’s Encrypt servers to have access to these folders:

# ee /usr/local/etc/apache24/httpd.conf

add the following:

# Lets Encrypt challenge directory configured per 
<Directory "/usr/local/www/.well-known/">
        Options None
        AllowOverride None
        Require all granted
        Header add Content-Type text/plain

And, for each VHOST that is going to get a cert:

# ee /usr/local/etc/apache24/extra/httpd-vhosts.conf

add to each non-ssl VHOST definition the following:

Alias /.well-known/ /usr/local/www/.well-known/

such that you end up with something like (yours may be different, especially watch out for BasicAuth or ModRewrite, addressed further down):

<VirtualHost IP.NU.MB.ER:80>
    DocumentRoot /usr/local/www/data-dist/domain-root
    ServerAlias *
    Alias /.well-known/ /usr/local/www/.well-known/
    ErrorLog /var/log/domain-error_log
    CustomLog /var/log/domain-access_log combined
    ScriptAlias /cgi-prg /www/cgi-prg

Don’t forget!

# apachectl restart

Step 4: First Try

At this point the system should be configured sufficiently to do a trial run with a single domain from the command line. Later on there are some scripts that will automate the process of both converting a large number of VHOSTed domains on a server to Let’s Encrypt and for maintaining them and getting email notifications if anything goes wrong in the, hopefully, fully automatic renewal process.

# acme-client -mvnNC /usr/local/www/.well-known/acme-challenge

This should create all the directories still needed and populate them, then check with the lets encrypt server and get a certificate and install it in the right place.  Inshalla.

If you get something like

acme-client: transfer buffer: [{ "type": "urn:acme:error:malformed","detail": "Provided agreement URL[]does not match current agreement URL[]","status": 400 }] (267 bytes)

That means the lets encrypt agreement has changed.  You can’t do much but write the port maintainer or wait for an update.  It will get fixed quickly and should only happen once a year.  I don’t think you’ll get it at all unless you’re unlucky enough to try to update when it is changing.  I was.

More likely you’ll get something like

acme-client: transfer buffer: [{ "type": "http-01", "status": "invalid", "error": { "type": "urn:acme:error:unauthorized", "detail": "Invalid response from (etc...)

This means there’s a problem accessing the /.well-known/ directory by the server.  There can be a lot of reasons for this:

  • You didn’t restart apache # apachectl restart
  • There was an error in the config file (look at the output of the restart) and therefore apache didn’t actually reaload with your new config.
  • DNS isn’t pointing where you think it is pointing.  Check with nslookup/whois to make sure.  Really.
  • You have the directories protected in some way – like with .htaccess.  (see below)

But if it goes well, you’ll get something like:

acme-client: /usr/local/etc/acme/ account key exists (not creating)
acme-client: /usr/local/etc/ssl/acme/private/ domain key exists (not creating)
acme-client: directories
acme-client: DNS:
acme-client: DNS: 2001:418:142b:290::3d5
acme-client: DNS: 2001:418:142b:28d::3d5
acme-client: req-auth:
acme-client: /usr/local/www/.well-known/acme-challenge/_ffVe6jHNHbIG1XKAeoqQmmtryWMGCKsfHIWWkl5lJw: created
acme-client: challenge
acme-client: status
acme-client: certificate
acme-client: full chain
acme-client: DNS:
acme-client: DNS:
acme-client: DNS: 2001:5a8:100::b817:9fb0
acme-client: DNS: 2001:5a8:100::b817:9fb1
acme-client: /usr/local/etc/ssl/acme/ created
acme-client: /usr/local/etc/ssl/acme/ created
acme-client: /usr/local/etc/ssl/acme/ created

Yay, you’ve got certs!  Now update your vhosts file to point to the certs you just created.  You may need to add a 443 container or, if it exists, update it to point to the new certs and restart apache.

# ee /usr/local/etc/apache24/extra/httpd-vhosts.conf

<VirtualHost IP.NU.MB.ER:443>
      DocumentRoot /usr/local/www/domainroot
      SSLCertificateFile /usr/local/etc/ssl/acme/
      SSLCertificateKeyFile /usr/local/etc/ssl/acme/private/
      SSLCertificateChainFile /usr/local/etc/ssl/acme/
      Header set Strict-Transport-Security "max-age=31536000; includeSubDomains"
      ErrorLog /var/log/domain-error_log
      CustomLog /var/log/domain-access_log combined

save and restart, look for any errors (typos on directory paths etc. will be detected and apache won’t restart, but be aware, it won’t quit either).

# apachectl restart
 Performing sanity check on apache24 configuration:
 Syntax OK
 Stopping apache24.
 Waiting for PIDS: 81160.
 Performing sanity check on apache24 configuration:
 Syntax OK
 Starting apache24.

Navigate to and check out your new green lock.  Check security and you should find:


Acme-Client Options

# man acme-client has all the deets, but we’re using:

  • -m to append the domain name to paths, use this and always use it or never.
  • -v for verbose output so we can see what is going on.
  • -n to check if an account key exists and create if not (no reason to omit)
  • -N to check if a domain key exists and create if not (also no reason to omit)
  • -C to specify the path to the challenge dir.  These guides all assume a centralized challenge dir outside the main serving path, and to which we redirect via an alias directive.
  • -F which forces the recreation of certs even if they haven’t expired (this counts against your 10 per 3 hours limit)
  • -s which redirects the process to the Let’s Encrypt staging server, which has no volume limits but also doesn’t create certs browsers accept.  (Using this is fine, but requires cleanup to switch to the production server, see below)
  • -e which is used to add a SAN to the certificate.  Removing one is a bit more involved (see below).

Automating Registration

Lets say you have a lot of domains, you might want to automate the process.  I modified the renewal script to automate the registration process.  This saved some time, but one quirk is you can only register 10 domains (certificates, including SANs, basically 10 lines of the domains list) per 3 hours (they say-I found it takes more like 12 hours to be allowed to register more).

First create  a file with all the domains you want to register for a Let’s Encrypt certificate in the same format as the renewal script uses (it can be the same file, but I made it different as I was experimenting)

# ee /usr/local/etc/acme/newdomains.txt 

# ee /usr/local/etc/acme/


# This script was adapted by Richard Fassett from
# by Bernard Spil
# See
# and updated again from richard fassett's script at
# this requires a file called /usr/local/etc/acme/newdomains.txt of the format
# domain.tld sub.domain.tld alt.domain.tld
# domain2.tld 
# domaind3.tld sub.domain3.tld 
# etc
# This should only be run to bulk-add domains.

# Define location of dirs and files

# Loop through the newdomains.txt file with lines like
cat ${DOMAINSFILE} | while read domain subdomains ; do

  # Create the cert directory with the command
  # acme-client -mvnNC /usr/local/www/.well-known/acme-challenge (domain subdomains)
  acme-client -mvnN -C "${CHALLENGEDIR}" ${domain} ${subdomains}


# chmod +x /usr/local/etc/acme/

A few fixes/recoveries that might be useful at this point: add SAN, remove SAN, switch from staging to production Let’s Encrypt servers.

Automation can break things, you might find you adjusted a few domains incorrectly or want to add a SAN later.

If you need to redo a domain from scratch, for example if you use the “s” option which created a cert from the staging server that doesn’t have volume limits (maybe you’re testing a lot of domains or trying to debug a particularly tricky .htaccess or DNS condition) – you might create a domain with acme-client -mvnsNC /usr/local/www/.well-known/acme-challenge and then want to generate the production cert.  You also need to do this to remove a SAN.  If you try without deleting the directories, you’ll get something like unknown SAN entry. (You replace “” with your domain.)

# setenv DD
# rm -r /usr/local/etc/ssl/acme/private/$DD && rm -r /usr/local/etc/acme/$DD && rm -r /usr/local/etc/ssl/acme/$DD && acme-client -mvnFNC /usr/local/www/.well-known/acme-challenge $DD www.$DD

If you need to add a new SAN to an existing domain

acme-client -mvneFNC /usr/local/www/.well-known/acme-challenge

it is the -e that “extends” the certificate.

Step 5: Automating Renewal

You might notice that the duration of the certificate is rather short: 3 months.  You really don’t want to be responding to certificate expired errors every 3 months, so let’s automate the renewal process.  For this you can create two files and store them on your server.  One is the renewal script itself and the other is a list of domains to renew.  This assumes you have more than one domain.  If you only have one domain, this is a bit overkill, but it will work, so why not?  You might get more domains in the future.   Everyone does.

First create a file with your list of domains, call it something creative like “domains.txt”  This is really a certificate request list with the “primary” domain and Subject Alternative Names (SANs) each on a single line.  In theory the SANs can be all over the place and Let’s Encrypt allows up to 100 per certificate (quite a lot), so the implication of “domains.txt” naming is a bit inaccurate, but that’s what everyone is using so we won’t be contrary.  You have to make sure that all the subdomains resolve—the Let’s Encrypt servers are going to look them up via DNS and if there aren’t working entries, this will fail with one of the errors above.  Check first.  I have not tested whether, if for example, you own,, and and they all point to the same directory, you can use one cert with different TLDs (or domains) as SANs; you should be able to, but I didn’t try.

# ee /usr/local/etc/acme/domains.txt

Now that you’ve saved that, the following script is adapted from a few at the references listed above and works on my server.  I made a few adjustments and corrections (there was a name change for acme-client which hasn’t quite propagated through all the HowTos yet).

# ee /usr/local/etc/acme/


# This script was adapted from by Bernard Spil
# See
# ... and further modified by David Gessel  
# This script will fail if the directories haven't been set up and
# domains in domain.txt have been successfully verified

# Define location of dirs and files

# is changed to 1 if any domains expired and were renewed

# Loop through the domains.txt file with lines like
cat ${DOMAINSFILE} | while read domain subdomains ; do

    # acme-client returns RC=2 when certificates 
    # weren't changed; use set +e to capture the return code
    set +e
    # Renew the key and certs if required
    acme-client -mvb -C "${CHALLENGEDIR}" ${domain} ${subdomains}
   # now that we have the return code, set script to exit if 
   # nonzero is returned
   set -e

   # if anything is expired, we'll want to do something 
   # (e.g., restart HTTPS)
   if [ $RC -ne 2 ] ; then

if [ "$CHECKEXPIRATION" -ne "0" ] ; then
        service apache24 restart

# chmod +x /usr/local/etc/acme/

This works quite well and will walk through your domains and renew as needed.

I have 36 domain/certificate lines in my “domains.txt” file and timing this script it takes 2.13 seconds to execute on my server.  There’s no real problem running it every night and if you have a lot of domains, you should remember you can only get 10 certs at a time and they won’t renew for about a week before expiry, a limitation I ran into in the bulk setup process.  You can spread your domain renewals out over the three months by force renewing blocks of them if you have more than about 60 per server.

You probably want to automate the process as a cron job. But before we do, lets address one more little problem: one of the shortcomings of the script process below is that the output messages of the script are output to stdout and only cron’s stderr is emailed to the admin. If your shell environment is wrong or the path to the script is wrong, cron will tell you, but if your domains don’t resolve or the script can’t reach /.well-known/, you will not get any warnings. That’s might be a bummer. So I redirect the output of the script to a log file. It gets overwritten with each execution, so it doesn’t need to be rotated – it is just the output of the last execution. It should be filled with lines including “adding SAN” (which it tells you for each domain) and “certificate valid” which it tells you for each cert that doesn’t need to be renewed. But it might tell you something else, like it barfed trying to reach the /.well-known/ directory because, say, you messed around with .htaccess or forgot to renew your domain and it is being redirected to parking or something. The following script first checks to see if there are any lines in /var/log/lets-encrypt-renew other than the expected, and if so, emails just those lines. You shouldn’t get anything until renewal time or if there’s an error. If you don’t care about renewal notices, you can edit the script to ignore those too.

# ee /usr/local/etc/acme/


# this script scans the log file created by the renewal execution cron job
# then removes any lines containing "adding SAN" or "certificate valid", which
# are normal messages, and mails whatever is left over using the "mail" command
# check full paths (or use relative) but full paths can avoid some errors
# use "# which grep" and "# which mail" on your system to check.


/usr/bin/grep -v "adding SAN" /var/log/lets-encrypt-renew | \
/usr/bin/grep -v "certificate valid" | /usr/bin/cat | \
{ while read status

  if [ "$PROBLEM" -ne "0" ] ; then
        /usr/bin/grep -v "adding SAN" /var/log/lets-encrypt-renew | \
        /usr/bin/grep -v "certificate valid" | \
        /usr/bin/mail -s "Lets Encrypt Errors" $1

# chmod +x /usr/local/etc/acme/

My cron configuration is set up as

# crontab -e

#*     *     *   *    *        command to be executed
#-     -     -   -    -
#|     |     |   |    |
#|     |     |   |    +----- day of week (0 - 6) (Sunday=0)
#|     |     |   +------- month (1 - 12)
#|     |     +--------- day of        month (1 - 31)
#|     +----------- hour (0 - 23)
#+------------- min (0 - 59)

# expanded path
# Let's Encrypt renewal check
*       3       *       *       *        /usr/local/etc/acme/ > & /var/log/lets-encrypt-renew
*       4       *       *       *        /usr/local/etc/acme/

Note that this requires that mail works. On servers that aren’t serving email, I use SSMTP and configured it more or less following this guide and and this especially the tip about using # chpass to change the default Full Name for root from “Charlie &” to something useful like “ServerName Root.”

You can test the mail function by adding a random word (or domain) to your domains.txt file and then executing

# /usr/local/etc/acme/ > & /var/log/lets-encrypt-renew
# /usr/local/etc/acme/

If everything is set up right, you’ll get an email complaining about your random word not being valid.  If you restore the correct domains.txt file and execute the above two commands you should not get an email at all.

# more /var/log/lets-encrypt-renew

should show only lines with “adding SAN” and “certificate valid” in them. If you execute # /usr/local/etc/acme/ you shouldn’t get any message.

.htaccess Problems

If you’re controlling access to a directory or have some non-HTML style process listening, you might run into challenges giving the Let’s Encrypt server access to the /.well-known/ directory.  I found the following formulation worked:

AuthType Basic
     AuthName "Please login."
     AuthUserFile "/xxx/.htpasswd"
     # the directive below also "requires" that the requested URL include /.well-known/
        Require expr %{REQUEST_URI} =~ m#^/.well-known/.*#
        Require valid-user

Basically the script above allows (requires) a “valid-user” (one with an entry in the AuthUserFile and valid matching password) and also requires (allows) a URL that is going to /.well-known/ and subdirectories thereof.  This also works in /usr/local/etc/apache24/httpd.conf and /usr/local/etc/apache24/extra/httpd-vhosts.conf

modRewrite to HTTPS problems

You can also create problems by rewriting to HTTPS.  You might want to do this now that you have certs that will auto-renew and you can provide a secure experience for everyone.   In order to get to the /.well-known/ directory, you have to add an exception to the mod-write rule for traffic to this subdirectory like so:

	RewriteEngine on
           RewriteCond %{REQUEST_URI} !^/\.well\-known/acme\-challenge/
	RewriteCond %{HTTPS} off
            RewriteRule (.*) https://%{SERVER_NAME}/$1 [R=301,L]

Also, if you redirect on a 404, some formulations cause problems. This one does not seem to:

ErrorDocument 404 /index.php

Posted at 06:43:58 UTC

Category: FreeBSDHowToSecuritytechnology

Open letter to the FCC 5 regarding net neutrality

Saturday, November 25, 2017 

I’m in favor of net neutrality for a lot of reasons; a personal reason is that I rely on fair and open transport of my bits to work overseas.  If you happen to find this little screed, you can also thank net neutrality for doing so as any argument for neutrality will likely be made unavailable by the ISPs that should charge exorbitant rents for their natural monopolies and would be remiss in their fiduciary responsibility should they fail to take every possible step to maximize shareholder value, for example by permitting their customers access to arguments contrary to their financial or political interests.

I sent the following to the FCC 5.  I am not, I’m sorry to say, optimistic.

Please protect Net Neutrality. It is essential to my ability to operate in Iraq, where I run a technical security business that relies on access to servers and services in the United States. If access to those services becomes subject to a maze of tiered access limitations and tariffs, rather than being treated universally as flat rate data, my business may become untenable unless I move my base of operations to a net neutrality-respecting jurisdiction. The FCC is, at the moment, the only bulwark against a balkanization of data and the collapse of the value premise of the Internet.

While I understand and am sympathetic to both a premise that less government regulation is better in principal and that less regulated markets can be more efficient; this “invisible hand” only works to the benefit in a “well regulated market.” There are significant cases where market forces cannot be beneficial, for example, where the fiduciary responsibility of a company to maximize share-holder value compels exploitation of monopoly rents to the fullest extent permitted by law and, where natural monopolies exist, only regulation prevents those rents from becoming abusive. Delivery of data services is a clear example of one such case, both due to the intrinsic monopoly of physical deployment of services through public resources and due to inherent opportunities to exert market distorting biases into those services to promote self-beneficial products and inhibit competition. That this might happen is not idle speculation: network services companies have routinely attempted to unfairly exploit their positions to their benefit and to the harm of fair and open competition and in many cases were restrained only by existing net neutrality laws that the FCC is currently considering rescinding. The consequences of rescinding net neutrality will be anti-competitive, anti-productive, and will stifle innovation and economic growth.

While it is obvious and inevitable that network companies will abuse their natural monopolies to stifle competition, as they have attempted many times restrained only by previous FCC enforcement of the principal of net neutrality, rescinding net neutrality also poses a direct risk to the validity of democracy. While one can argue that Facebook has already compromised democracy by becoming the world’s largest provider of news through an extraordinarily easily manipulated content delivery mechanism, there’s no evidence that they have yet exploited this to achieve any particular political end nor actively censored criticism of their practices. However, without net neutrality there is no legal protection to inhibit carriers from exploiting their control over content delivery to promote their corporate or political interests while censoring embarrassing or opposing information. As the vast majority of Americans now get their news from on-line resources, control over the delivery of those resources becomes an extraordinarily powerful political weapon; without net neutrality it is perfectly legal for corporations to get “their hands on those weapons” and deploy them against their economic and political adversaries.

Under an implicit doctrine of net neutrality from a naive, but then technically accurate, concept of the internet as a packet network that would survive a nuclear war and that would treat censorship as “damage” and “route around it automatically,” to 2005’s Madison River ruling, to the 2008 Comcast ruling, to 2010’s Open Internet Order the internet has flourished as an open network delivering innovative services and resources that all businesses have come to rely on fairly and equally. Overturning that historical doctrine will result in a digital communications landscape in the US that resembles AT&Ts pre-breakup telephone service: you will be permitted to buy only the services that your ISP deems most profitable to themselves. In the long run, if net neutrality is not protected, one can expect the innovation that has centered in the US since the birth of the internet, which some of us remember as the government sponsored innovation ARPAnet, to migrate to less corporatist climates, such as Europe, where net neutrality is enshrined in law.

The American people are counting on you to protect us from such a catastrophic outcome.

Do not reverse the 2015 Open Internet Order.


David Gessel


Posted at 10:27:50 UTC

Turn off windows update now!

Monday, March 14, 2016 

If you haven’t already, turn off Windows update now.  Microsoft has recently started installing Windows 10 spyware without consent.  A good friend of mine had a bunch of systems at the company where he runs IT hacked by Microsoft over the weekend, which broke the certificate store for WPA-2 and thus their wifi connections.

To be clear, Windows 10 is spyware.  Microsoft has changed their business model from selling a product to selling data – your data – to whoever they want.  Windows 10 comes with a EULA that gives them the right to steal everything on your computer – your email, your private pictures, your home movies, your love letters, your medical records, your financial records – anything they want without telling you.  “If you’re not paying for the product, you are the product.

If this happens to you,  I suggest contacting your state attorney general and filing a complaint against Microsoft.  Hopefully a crushing class action suit or perhaps jail time for the executives that dreamed up this massive heist will help deter future corporate data thieves, though that’s certainly irrational optimism.

I wish I could recommend switching to Linux for everyone, but there’s a lot of software that still depends on Windows and a lot of users that will have a hard time migrating (developers: please stop developing for Microsoft).  Apple seems unequivocally better in refusing to act as key player in bringing about Total Information Awareness.  I’m not a huge fan of their walled garden and computers as overpriced fashion accessories approach, but it is far better than outright theft.  For those that are slightly computer savvy, there’s Linux Mint, which is quite usable and genuinely free.

These instructions might help prevent that disaster of an update being visited upon you (and possibly law enforcement visits to come after Microsoft starts sifting through all your datas and forwarding on whatever they find).  The latest reports suggest they aren’t enough, but it is the best I have found other than isolating your windows box from the internet completely.

Posted at 14:27:03 UTC

Category: NegativePrivacySecuritytechnology

PGP Usability Regression thanks to Enigmail

Thursday, February 25, 2016 

The latest auto update to Enigmail, the essential plugin for Thunderbird for encrypted mail, is a fairly dynamic project that occasionally makes UI and usability decisions that not everyone agrees with.

The latest is a problem for me.  I use K9 for mobile mail and K9 doesn’t support PGP/MIME, but Enigmail just:


Why?  OK – PGP/MIME leaks less metainformation than inline PGP, but at the expense of compatibility.  K9 should support PGP/MIME, but it doesn’t.  Enigmail should have synchronized with K9 and released PGP/MIME when mobile users could use it.

But encryption people often insist that the only use case that matters is some edge case they think is critical.  They like to say that nobody should read encrypted mail on a mobile device because the baseband of the device is intrinsically insecure (all cell phones are intrinsically insecure – phones should treat the data radio as a serial modem and the OS and the data modem should interact only over a very simple command set – indeed, the radio should be a replaceable module, but that gets beyond this particular issue).

For now, make sure your default encoding is Inline-PGP or you’ll break encryption.   Encryption only works if it is easy to use and universally available. When people can’t read their messages, they just stop using it.  This isn’t security, this is a mistake.

Posted at 01:52:42 UTC

Category: cell phonesPrivacySecuritytechnology

Signal Desktop: Probably a good thing

Tuesday, December 8, 2015 

Signal is an easy to use chat tool that competes (effectively) with What’sApp or Viber. They’ve just released a desktop version which is being “preview released/buzz generating released.”  It is developed by a guy with some cred in the open source and crypto movement, Moxie Marlinspike.  I use it, but do not entirely trust it.

I’m not completely on board with Signal.  It is open source, and so in theory we can verify the code.  But there’s some history I find disquieting.  So while I recommend it as the best, easiest to use, (probably) most secure messaging tool available, I do so with some reservations.

  • It originally handled encrypted SMS messages.  There is a long argument about why they broke SMS support on the mailing lists.  I find all of the arguments Whisper Systems made specious and unconvincing and cannot ignore the fact that the SMS tool sent messages through the local carrier (Asiacell, Korek, or Zain here).  Breaking that meant secure messages only go through Whisper Systems’ Google-managed servers where all metadata is captured and accessible to the USG. Since it was open source, that version has been forked and is still developed, I use the SMSSecure fork myself
  • Signal has captured all the USG funding for messaging systems.  Alternatives are not getting funds.  This may make sense from a purely managerial point of view, but also creates a single point of infiltration.  It is far easier to compromise a single project if there aren’t competing projects.   Part of the strength of Open Source is only achieved when competing development teams are trying to one up each other and expose each other’s flaws (FreeBSD and OpenBSD for example).  In a monoculture, the checks and balances are weaker.
  • Signal has grown more intimate with Google over time.  The desktop version sign up uses your “google ID” to get you in the queue.  Google is the largest commercial spy agency in the world, collecting more data on more people than any other organization except probably the NSA.  They’re currently an advertising company and make their money selling your data to advertisers, something they’re quite disingenuous about, but the data trove they’ve built is regularly mined by organizations with more nefarious aims than merely fleecing you.

What to do?  Well, I use signal.  I’m pretty confident the encryption is good, or at least as good as anything else available.  I know my metadata is being collected and shared, but until Jake convinces Moxie to use anonymous identifiers for accounts and message through Tor hidden nodes, you have to be very tech savvy to get around that and there’s no Civil Society grants going to any other messaging services using, for example, an open standard like a Jabber server on a hidden node with OTR.

For now, take a half step up the security ladder and stop using commercial faux security (or unverifiable security, which is the same thing) and give Signal a try.

Maybe at some later date I’ll write up an easy to follow guide on setting up your own jabber server as a tor hidden service and federating it so you can message securely, anonymously, and keep your data (meta and otherwise) on your own hardware in your own house, where it still has at least a little legal protection.


Posted at 10:21:22 UTC

Category: PositivePrivacyreviewsSecuritytechnology

10 Gbyte Win10 Spyware “upgrade” now forced on users

Sunday, September 27, 2015 

Microsoft has, historically, done some amazingly boneheaded things like clippy, Vista, Win 8, and Win 10.  They have one really good product: Excel, otherwise everything they’ve done has succeeded only through illegal exploitation of an aggressively defended monopoly. OK, maybe the Xbox is competitive, but I’m not much of a gamer.

Sadly for the world, the model of selling users for profit to advertisers and spies has gained ground to the point where Microsoft was starting to look like the least evil major entity in closed-source computing.  Poor microsoft.  To lose the evil crown must be at least as humiliating as their waning revenue and abject failures in the mobile space (so strange… try to enter a space where they don’t have a monopoly to force users to accept their mediocre crap and they fail, who’da thunk it?)

“There is a difference between policy and practice. We don’t read customers mail. We don’t read customer documents. We don’t triangulate YouTube views and searches. We don’t use the content of your Hotmail to target ads in Bing,”

Frank Shaw, Corporate Vice President of Corporate Communications for Microsoft

Well, never fear: Windows 10 is here and they’re radically one-upping the data theft economy by p0wning not just the data you idiotically entrust to someone else’s server for free without ever considering why they’re giving you that useful service for “free” or what they, or whoever buys their ultimately failed business, might do with your data, but also the data you consider too sensitive for the Google or the Apple.  Windows 10 exfiltrates all your data to Microsoft for their use and profit without your information.  Don’t believe it? Read their Privacy Statement.

Finally, we will access, disclose and preserve personal data, including your content (such as the content of your emails, other private communications or files in private folders), when we have a good faith belief that doing so is necessary.

And it is free (as in beer but not as in speech).  What could possiblay go wrong?

Well, people weren’t updating fast enough so Microsoft is now pushing that update on you involuntarily.  Do you have a data cap that a 10G download might break and cost you money?  So what!  Your loss!  Don’t have enough space on your drive for a 10G hidden folder of crapware foisted off on you without your permission?  Tough crap, Microsoft don’t care.

To be clear, Windows 10 is spyware.  If this was coming from a teenage hacker somewhere, they’d be facing jail time.  It is absolutely, unequivocally malware that will create a liability for you if you use it.  If you have any confidentiality requirement, you must not install windows 10.  Ever. Not even on your home machine.  Just don’t.

The only way to prevent this is really annoying and a little risky: disable automatic downloads.  One of the problems with Microsoft’s operating systems is the unbelievably crappy spaghetti code that results in a constant flow of cracks, a week’s worth are patched every Tuesday.  About 1 serious vulnerability every fortnight these days (note this is about the same as Ubuntu and about 1/4 the rate of OSX or iOS, why people think Apple products are “secure” is beyond me – live in that fantasy walled garden!  But nice logo you paid a 50% premium for on your shiny device). Not patching increases the risk that some hacker somewhere will steal your datas, but patching guarantees that Microsoft will steal your datas.  Keep your anti-virus up to date and live a little dangerously by keeping Microsoft out.

Here’s an interesting article: how-to-clean-the-windows-10-crapware-off-your-windows-7-or-81-pc

And a tool referenced in that article: GWX control panel (that can help remove the windows 10 infection if you got it).

And a list of patches I found that are related to Win10 malware that you can remove if you haven’t installed it yet (Windows 10 eliminates the ability to choose or selectively remove patches, once you’re in for the ride, you’re chained in: all or nothing.)

Basic advice:

  • Disable automatic updates and automatic downloads of updates.
  • Review each update Microsoft offers.  This is tedious, my win 7 install reports 384 updates, 5-10 a week, but other than security patches, you probably don’t really need them.  Only install a patch if there’s a reason.  Sorry, that sucks, but there’s always Linux Mint: free like beer AND free like speech.
  • If you’re still on Win 7/8, uninstall the spyware Microsoft has probably already installed.  If you’re on Windows 8, you probably want to upgrade to Windows 7 if at all possible.
  • If you succumbed to the pressure and became a Microsoft Product by installing Windows 10, uninstall it.
  • If uninstall doesn’t work, switch to Mint or reinstall 7.

Most importantly, if you develop software for servers or for end users, stop developing for Microsoft (and Apple too).  Respect the privacy of your customers by not exposing them to exploitation by desperate operating system vendors.  In many classes of applications, your customers buy their computers to run your software: they don’t care what operating system it requires – that should be transparent and painless.  Microsoft is no longer an even remotely acceptable choice.  Server applications should run under FreeBSD or OpenBSD and desktop applications should run under Linux.  You can charge more and generate more profit because the total net cost for your customers will be lower.  Split the difference and give them a more reliable, more secure, and lower cost environment and make more money doing so.

Posted at 08:07:54 UTC

Category: FreeBSDHowToLinuxSecuritytechnology

The CA System is Intractably Broken

Tuesday, July 21, 2015 

I’m dealing with the hassle of setting up certs for a new site over the last few days. It means using startcom’s certs because they’re pretty good (only one security breach) and they have a decently low-hassle free certificate that won’t trigger BS warnings in browsers marketing fake cert mafia placebo security products to unwitting users. (And the CTO answers email within minutes well past midnight.)

And in the middle of this, news of another breach to the CA system was announced on the heels of Lenovo’s SuperFish SSL crack, this time a class break that resulted in a Chinese company being able to generate the equivalent of a lawful intercept cert and provided it to a private company. Official lawful intercept certificates are a globally used tool to silently crack SSL so official governments can monitor SSL encrypted traffic in compliance with national laws like the US’s CALEA.

But this time, it went to a private company and they were using it to intercept and crack Google traffic, and Google found out. The absurdity is to presume that this is an infrequent event. Such breaches (and a “breach” isn’t a lawful intercept tool, which are in constant and widespread use globally, but such a tool in the “wrong” hands) happen regularly. There’s no data on the ratio of discovered breaches to undiscovered breaches, of course. While it is possible that they are always found, seemingly accidental discoveries suggest far wider misuse than generally acknowledged.

The cert mafia should be abolished. Certificate authorities work for authoritarian environments in which a single entity is trusted by fiat as in a dictatorship or a company. The public should trust public opinion and a tool like Perspectives would end these problems as well as significantly lower the barrier to a fully encrypted web as those of us trying to protect our traffic wouldn’t need to choose between forking over cash to the cert mafia for fake security or making our users jump through scary security messages and complex work-arounds.

Posted at 00:53:59 UTC

Category: FreeBSDPrivacySecuritytechnology

A sad loss for security

Monday, July 20, 2015 

Whisper systems wrote the very useful TextSecure app for Android. It had a great feature of encrypting text messages, a standard communication modality in much of the world and one I rely on often. I have previously suggested it is a good tool.

The last “update” removed the ability to establish new encrypted chats over SMS and, it appears, the next will remove the function entirely. For me, this change substantially reduces the utility of the app.

Reading their arguments for doing so, I find myself disagreeing with their justifications. I understand there was some complexity in establishing encrypted SMS, but frankly initiating a one-time key exchange was about as easy as encrypted communication gets. That iOS users can’t play along is pretty irrelevant: iOS isn’t exactly the platform for secure communications anyway, you carry iOS devices when you want to impress people with your brand awareness, not get things done. That people occasionally end up with a conversation that is half-encrypted seems annoying but hardly all that problematic. The person that uninstalled the app will try to send messages in the clear, not the person who is still running it and a partial session. I can see the annoyance, but not any security leak.

I think the final result is somewhat dangerous. The first incarnation used SMS as the starting point, and once a secure communications were established, if available, coms moved transparently to the data channel. If not, it stayed with SMS. As I work in a place where data service is frequently disabled, this was the most reliable non-voice communication protocol.

Now SMS is unencrypted and data-mode communication is encrypted. You have to remember which is which and that is dangerous.

If they don’t restore encrypted SMS functionality, I will switch back to the standard SMS app, which is insecure SMS only and so not confusing and use chat secure or xabber for encrypted data communications so the difference is clear. You’re probably going to run a jabber-based chat tool anyway chat secure’s Tor integration makes it a better choice for data-mode chat while text secure no longer does anything particularly useful over the default app for SMS-mode nor anything particularly unique for data mode.

Posted at 00:53:41 UTC

Category: cell phonesSecurity

Making Chrome Less Horrible

Saturday, June 13, 2015 

Google’s Chrome is  a useful tool to have around, but the security features have gotten out of hand and make it increasingly useless for real work without actually improving security.

After a brief rant about SSL, there’s a quick solution at the bottom of this post.


Chrome’s Idiotic SSL Handling Model

I don’t like Chrome nearly as much as Firefox,  but it does do some things better (I have a persistent annoyance with pfSense certificates that cause slow loading of the pfSense management page in FF, for example). Lately I’ve found that the Google+ script seems to kill firefox, so I use Chrome for logged-in Google activities.

But Chrome’s handling of certificates is abhorrent.  I’ve never seen anything so resolutely destructive to security and utility.  It is the most ill-considered, poorly implemented, counter-productive failure in UI design and security policy I’ve ever encountered.  It is hateful and obscene.  A disaster.  An abomination. The ill-conceived excrement of ignorant twits.  I’d be happy to share my unrestrained feelings privately.

It is a private network, you idiots

I’ve discussed the problem before, but the basic issues are that:

  • The certificate authority is NOT INVALID, Chrome just doesn’t recognize it because it is self-signed.  There is a difference, dimwits.
  • This is a private network (10.x.x.x or 192.168.x.x) and if you pulled your head out for a second and thought about it, white-listing private networks is obvious.  Why on earth would anyone pay the cert mafia for a private cert?  Every web-interfaced appliance in existence automatically generates a self-signed cert, and Chrome flags every one of them as a security risk INCORRECTLY.
  • A “valid” certificate merely means that one of the zillions of cert mafia organizations ripping people off by pretending to offer security has “verified” the “ownership” of a site before taking their money and issuing a certificate that placates browsers
  • Or a compromised certificate is being used.
  • Or a law enforcement certificate is being used.
  • Or the site has been hacked by criminals or some country’s law enforcement.
  • etc.

A “valid” certificate doesn’t mean nothing at all, but close to it.

So one might think it is harmless security theater, like a TSA checkpoint: it does no real harm and may have some deterrent value.  It is a necessary fiction to ensure people feel safe doing commerce on the internet.  If a few percent of people are reassured by firm warnings and are thus seduced into consummating their shopping carts, improving ad traffic quality and thus ensuring Google’s ad revenue continues to flow, ensuring their servers continue sucking up our data, what’s the harm?

The harm is that it makes it hard to secure a website.  SSL does two things: it pretends to verify that the website you connect to is the one you intended to connect to (but it does not do this) and it does actually serve to encrypt data between the browser and the server, making eavesdropping very difficult.  The latter useful function does not require verifying who owns the server, which can only be done with a web of trust model like perspectives or with centralized, authoritarian certificate management.

How to fix Chrome:

The damage is done. Millions of websites that could be encrypted are not because idiots writing browsers have made it very difficult for users to override inane, inaccurate, misleading browser warnings.  However, if you’re reading this, you can reduce the headache with a simple step (Thanks!):

Right click on the shortcut you use to launch Chrome and modify the launch command by adding the following “--ignore-certificate-errors

Unfuck chrome a bit.

Once you’ve done this, chrome will open with a warning:

zomg: ignore certificate errors?  who doesn't anyway?

YAY.  Suffer my ass.

Java?  What happened to Java?

Bonus rant

Java sucks so bad.  It is the second worst abomination loosed on the internet, yet lots of systems use it for useful features, or try to.  There’s endless compatibility problems with JVM versions and there’s the absolutely idiotic horror of the recent security requirement that disables setting “medium” security completely no matter how hard you want to override it, which means you can’t ever update past JVM 7.  Ever.  Because 8 is utterly useless because they broke it completely thinking they’d protect you from man in the middle attacks on your own LAN.

However, even if you have frozen with the last moderately usable version of Java, you’ll find that since Chrome 42 (yeah, the 42nd major release of chrome. That numbering scheme is another frustratingly stupid move, but anyway, get off my lawn) Java just doesn’t run in chrome.  WTF?

Turns out Google, happy enough to push their own crappy products like Google+, won’t support Oracle’s crappy product any more.  As of 42 Java is disabled by default.  Apparently, after 45 it won’t ever work again.  I’d be happy to see Java die, but I have a lot of infrastructure that requires Java for KVM connections, camera management, and other equipment that foolishly embraced that horrible standard.  Anyhow, you can fix it until 45 comes along…

To enable Java in Chrome for a little while longer, you can follow these instructions to enable NPAPI (which enables Java).  Type “chrome://flags/#enable-npapi” in the browser bar and click “enable.”

Enable NAPI

Posted at 13:24:37 UTC

Category: HowToSecuritytechnology

Superfish proves certs are useless for identification

Saturday, February 21, 2015 

Can we please, please stop with the stupid certificate verification warnings?

superfish logo

Dear security developers, your model is broken. It never worked. Stop warning people about certificate errors. Now. Forever.

Certificate errors serve two purposes:

  1. They make developers uncomfortable with using perfectly secure self-signed certs, and since commercial certs cost money, much of the web that could be encrypted remains unencrypted. That’s harm done to the public. Thanks.
  2. They happen so often, so relentlessly, for such trivial reasons (not even Google can keep their certs up to date) that users learn to ignore them, which makes an actual man-in-the-middle attack almost certain to succeed with most people, despite the warnings.

The Certificate Authority system is predicated on the idea that Certificate Authorities are flawless and trustworthy. They are neither. The Lenovo/Superfish problem shows another obvious flaw: hardware vendors (and actually any trusted software installer) has to be trustworthy too or client-side MITM is easy. And CA’s simply can’t verify against that.

This whole idiocy creates massive problems for something so basic as LAN administration. Even before wireless became pervasive, LAN coms should always be encrypted when passwords or any meaningful data is moving. Current security settings create a massive avalanche of useless errors for “untrustworthy certs” on one’s own network (the obvious fix is to automatically trust all certs on private networks, duh).

This is an issue that bothers me a lot. It gets in my way constantly and makes real security and encrypted communications way harder and way more complicated than it needs to be and the only beneficiaries at all are the certificate Mafiosi. This is just stupid. Superfish proves, again, how broken it is. Can we stop pretending now?

Also, this most recent of many certificate flaws comes with a bonus feature: the MITM cert Superfish uses is apparently really pathetically insecure, aside from using broken crypto, their software had their passwords in it, making it easy for crackers to develop tools to harvest additional data from the victims of the Superfish/Lenovo attack.It probably hurts more to find out your vendor hacked you, but the penalty is that the hack also destroyed the security of all of your communications. Thanks. This is why we can’t have nice things. It is also why any back door, no matter what the motive, compromises security.


Update: Superfish is, apparently, out of business.  While that sucks for the people at the company, who were probably very happy with their Lenovo OEM deal and instead got a big sock of coal, one might naively hope for an upside that companies considering a model based on stealing people’s data might take notice of the cautionary tale of superfish.

Unfortunately – that won’t happen, not in the current valley climate. While it is economically advantageous to hire cheap kids who have no life and will work long hours for meagre pay, they come with a downside: they are all ignorant idiots. I don’t mean they’re not smart or capable (though the smart barrel was long ago drained and the vast majority of brogrammers sauntering around SF really are stupid), rather that they are foolish as in the opposite of ‘wise.”  Wisdom comes from experience, and experience only comes with time, an immutable dimension.  This superfish debacle was only from Feb 2015, but this year’s batch of idiot brogrammers weren’t around to see it and as they gather in self-congratulatory clusters in posh, VC-funded collaborative spaces, company barrista-brewed latte in one hand and social-media-distraction feeding portable device in the other, they’ll be high-fiveing and fist-bumping the brilliance of their brand-new idea for getting around SSL so they can collect marketing data and better target advertising.  Yay.


How to fix Superfish:

Install Perspectives. And support them.

Also, this bugs the crap out of me:

Overthrow the Cert Mafia!

SSL for Authentication Sucks

Unbreaking Firefox SSL Behavior

The CA System is Intractably Broken

Posted at 02:45:41 UTC

Category: Securitytechnology