Geopost
Posts that include geographic metadata. Or posts about GIS/Geographic systems.
Manually Update Time Zone Data on Android 10
One of the updates that stops when your carrier decides you have to buy a new phone to keep their profits up is the time zone data, which means as regions decide they will or won’t continue using standard time and will switch permanently to lazy people time (or not), time zone calculations start to fail, which can be awfully annoying when it causes you to miss flights or meetings. It is probably something you’ll want to keep up to date. Unfortunately, this requires root access to your phone because… profits depend on the velocity by which first world money is converted to e-waste to poison third world children. Yay.
Root requires reflashing your device, which means wiping all your data and apps and reinstalling them, so easier to do on a new phone than backing up and restoring and re-configuring all your apps. Sooner or later your vendor will stop supporting your device in an attempt to get you to throw it away and buy a new one and you’ll have to root it to keep it up to date and secure so you might as well do it now, void their stupid warranty, and take control of your device.
You should also take a moment to write your elected representatives and demand that they take civil action against this crap. Lets take a short rant break, shall we?
Planned obsolescence, death by security flaws, and vendor locks should be prosecuted, not just as illegal profiteering but as environmental crimes for needlessly flooding the world with e-waste. If you own a device you have the right to use it as you like and any entity that by omission or obfuscation of reasonable information needed to keep that device operational is depriving legitimate owners of rightful value. Willfully obstructing security updates, knowing full well the risks implied, is coercive if not extortion. Actively blocking the provision of third party services intended to mitigate these harms through barratry and legal extortion should be prosecuted aggressively. Everyone who has purchased a phone that has been intentionally and unfairly life-limited by non-replaceable batteries, intimidation of repair services, manipulation of the spare parts market, or restrictions or obfuscation of security updates is due refund of the value thus denied plus penalties.
Ah, that feels better, no?
Assuming you have a rooted phone, adb installed on your computer, and your TZ data is out of date, lets get it fixed, shall we? The problem is that TZ data comes from IANA, from here actually, and is versioned in a form like 2023c, the current as of now. That’s lovely but the format they provide is not compatible with android and needs to be transformed. Google seems to have some tools for this in the FOSS branch of Android, but it seems a little useless without a virtual environment, a PITA. But the good folks at LineageOS (yay, FOSS!!!) maintain their version of the tool with the thus created output data in their git, which we can use for all android devices (it seems). The files we need are in this directory: note that these are 2023a, but 2023c is identical to 2023a, reverting some changes made in 2023b because, I don’t know, the whole mess about getting up an hour earlier or later being some traumatic experience when it happens twice a year is catastrophic for people’s sense of well being, but when they get up at different times on days off than on work days, that doesn’t count or something. OMG. so drama. people. sometimes it hurts to be associated with them as a species. Not that I care, but stop messing around and just pick one. So many rant triggers in this whole mess.
Anyway, proceeding with the assumption your device is rooted and you have adb installed on your computer, the files needed are:
tzdata a binary file that if you view with a text editor should start with: tzdata2023a tzlookup.xml an xml file that should (nearly) start with: <timezones ianaversion="2023a"> tz_version a simple text file that should have one line: 003.001|2023a|001
Download the compressed .tgz archive of the output_data
directory from here by clicking on the [tgz]
text at the top right
You should get a .tgz archive, from which you want to extract:
tzlookup.xml
from theandroid
foldertzdata
from theiana
foldertz_version
from theversion
folder
Here’s the tricky bit, you gotta get these files to the right places. So I mounted my android on my computer and created a folder TZData
in Downloads
and copied the files there, this resolved to /data/media/0/Download/TZdata/
on my device. While you’re there, make a folder like oldTZ
in the same place for backup. Everything else is done by command line via adb.
(comments are demarked with "#", the prompt is assumed) # get shell on your device adb shell # get root, if this fails, you don't have root, bummer, you don't really own your device. su root # verify your tz data is where mine was, if so copypasta should be safe. find / -name tzdata 2>/dev/null #output for me looks like some are symlinks /apex/com.android.tzdata/etc/tz/tzdata /apex/com.android.tzdata@290000000/etc/tz/tzdata /apex/com.android.runtime/etc/tz/tzdata /apex/com.android.runtime@1/etc/tz/tzdata /system/apex/com.android.runtime.release/etc/tz/tzdata /system/apex/com.android.tzdata/etc/tz/tzdata /system/usr/share/zoneinfo/tzdata # did ya get the same or close enough to figure out what to do next? good. # Backup your old stuff cp /system/apex/com.android.tzdata/etc/tz/* /data/media/0/Download/oldTZ # your directories are read only, so you need to fix that, scary but reversible mount -o rw,remount / mount -o rw,remount /apex/com.android.tzdata mount -o rw,remount /apex/com.android.runtime # copy the new files over the old files, the last location is legacy and doesn't # seem to have a copy of tzlookup.xml, so we don't put a new one there, but check ls /system/usr/share/zoneinfo # only tzdata and tz_version? Good. cp /data/media/0/Download/TZdata/* /apex/com.android.tzdata/etc/tz cp /data/media/0/Download/TZdata/* /apex/com.android.runtime/etc/tz cp /data/media/0/Download/TZdata/* /system/apex/com.android.tzdata/etc/tz cp /data/media/0/Download/TZdata/tz_version /system/usr/share/zoneinfo cp /data/media/0/Download/TZdata/tzdata /system/usr/share/zoneinfo # all done, now we just gotta read-only those directories again mount -o ro,remount / mount -o ro,remount /apex/com.android.tzdata mount -o ro,remount /apex/com.android.runtime # and why not reboot from the command line? reboot
That was fairly painless once you know what to do and have root, no? it worked for me, my phone rebooted and the time zone database appears to be updated. YMMV, hopefully not the reboot successfully part but bricking a phone is a risk because, you know, profits. After that tz file surgery I created a new event in a US time zone that recently changed their daylight savings to pacify the crazies and it seemed to work as expected.
Some gnuplot and datamash adventures
I’ve been collecting data on the state of the Ukrainian digital network since about the start of the war on a daily basis, some details of the process are in this post. I was creating and updating maps made with qgis when particularly notable things happened, generally correlated with significant damage to the Ukrainian power infrastructure (and/or data infrastructure). I wanted a way to provide a live update of the feed, and as all such projects go, the real reward was the friends made along the way to an automatically updated “live” summary stats table and graph.
My data collection tools generate some rather large CSV files for the mapping tools, but to keep a running summary, I also extract the daily total of responding servers and compute the day over day change and append those values to a running tally CSV file. A few really great tools from the Free Software Foundation help turn this simple data structure into a nicely formatted (I think) table and graph: datamash and gnuplot. I’m not remotely expert enough to get into the full details of these excellent tools, but I put together some tricks that are working for me and might help someone else trying to do something similar.
Using datamash for Statistical Summaries
Datamash is a great command line tool for getting statistics from text files like logs or CSV files or other relatively accessible and easily managed data sources. It is quite a bit easier to use and less resource intensive than R, or Gnu Octave, but obviously also much more limited. I really only wanted very basic statistics and wanted to be able to get to them from Bash
with a cron
job calling a simple script and for that sort of work, datamash
is the tool of choice.
Basic statistics are easy to compute with datamash
; but if you want a thousands grouped comma delimited median value of a data set that looks like 120,915
(say), you might need a slightly more complicated (but still one-liner) command like this:
Median="$(/usr/bin/datamash -t, median 2 < /trend.csv | datamash round 1 | sed ':a;s/\B[0-9]\{3\}\>/,&/;ta')" Median= Assign the result to the variable $Median -t, Comma delimited (instead of tab, default) median one of a bazillion stats datamash can compute 2 use column two of the CSV data set. < /trend.csv feed the previous command a CSV file nom nom | datamash round 1 pipe the result back to datamash to round the decimals away | sed (yadda yadda) pipe that result to sed to insert comma thousands separator*
Once I have these values properly formatted as readable strings, I needed a way to automatically insert those updates into a consistently formatted table like this:
I first create a dummy table with a plugin called TablePress with target dummy values (like +++Median
) which I then extract as HTML and save as a template for later modification. With the help of a little external file inclusion code into WordPress, you can pull that formatted but now static HTML back into the post from a server-side file. Now all you need to do is modify the HTML file version of the table using sed
via a cron
job to replace the dummy values with the datamash
computed values and then scp
the table code with updated data to the server so it is rendered into the viewed page:
sed -i -e "s/+++Median/$Median/g" "stats_table.html" /usr/bin/sshpass -P assphrase -f '~/.pass' /usr/bin/scp -r stats_table.html user@site.org:/usr/local/www/wp-content/uploads/stats_table.html
For this specific application the bash
script runs daily via cron
with appropriate datamash
lines and table variable replacements to keep the table updated on a daily basis. It first copies the table template into a working directory, computes the latest values with datamash
, then sed
s those updated values into the working copy of the table template, and scp
s that over the old version in the wp-content
directory for visitor viewing pleasure.
Using gnuplot for Generating a Live Graph
The basic process of providing live data to the server is about the same. A different wordpress plugin, SVG Support, adds support for SVG filetypes within WordPress. I suspect this is not default since svg can contain active code, but a modern website without SVG support is like a fish without a bicycle, isn’t it? SVG is useful in this case in another way, the summary page integrates a scaled image which is linked to the full size SVG file. For bitmapped files, the scaled image (or thumbnail) is generated by downsampling the original (with ImageMagick, optimally, not GD) and that needs an active request (i.e. PHP code) to update. In this case, there’s no need since the SVG thumbnail is the just the original file resized—SVG: Scalable Vector Graphics FTW.
Gnuplot
is a impressively full-featured graphing tool with a complex command structure. I had to piece together some details from various sources and then do some sed
ding to get the final touches as I wanted them. As every plot is different, I’ll just document the bits I pieced together myself, the plotting details go in the gnuplot command script, the other bits in a bash script executed later to add some non-standard formatting to the gnuplot svg output.
Title of the plot
The SVG <title>
block is set as “Gnuplot” and I don’t see any way to change that from the command line, so I replaced it with the title I wanted, using a variable for the most recently updated data point extracted by datamash
as above:
sed -i -e "s/<title>Gnuplot<\/title>/<title>Ukrainian Servers Responding on port 80 from 2022-03-05 to $LDate<\/title>/g" "/UKR-server-trend.svg" sed -i -e "s/<desc>Produced by GNUPLOT 5.2 patchlevel 2 <\/desc>/<desc>Daily automated update of Ukrainian server response statistics.<\/desc>/g" "/UKR-server-trend.svg"
This title value is used as the tab title. I’m not sure where the <desc>
will show up, but likely read by various spiders and is an accessibility thing for online readers.
Last Data Point
I wanted the most recent server count to be visible at the end of the plot. This takes two steps: first plot that data point alone with a label (but no title so it doesn’t show up in the data key/legend) by adding a separate plot of just that last datum like:
"< tail -n 1 '/trend.csv'" u 1:2:2 w labels notitle
This works fine, but if you hover over the data point, it just pops up “gnuplot_plot_4” and I’d rather have more useful data so I sed
that out and replace it with some values I got from datamash
queries earlier in the script like so:
sed -i -e "s/<title>gnuplot_plot_4<\/title>/<title>Tot: $LTot; Diff: $LDif<\/title>/g" "/UKR-server-trend.svg"
Adding Link Text
SVG supports clickable links, but you can’t (I don’t think) define those URLs in the label command. So first set the visible text with a simple gnuplot label command:
set label "Black Rose Technology https://brt.llc" at graph 0.07,0.03 center tc rgb "#693738" font "copperplate,12"
and then enhance the resulting svg code with a link using good old sed
:
sed -i -e "s#<text><tspan font-family=\"copperplate\" >Black Rose Technology https://brt.llc</tspan></text>#<a xlink:href=\"https://brt.llc/\" target=\"__blank\"><text><tspan font-family=\"copperplate\" >Black Rose Technology https://brt.llc</tspan></text></a>#g" "/UKR-server-trend.svg"
Hovertext for the Delta Bars
Adding hovertext to the ends of the daily delta bars was a bit more involved. The SVG <title>
type is interpreted by most browsers as a hoverable element but adding visible data labels to the ends of the bars makes the graph icky noisy. Fortunately SVG supports transparent text. To get all this to work, I replot the entire bar graph data series as just labels like so:
'/trend.csv' using 1:3:3 with labels font "arial,4" notitle axes x1y2
But this leaves a very noisy looking graph, so we pull out our trusty sed
to set opacity to “0
” so they’re hidden:
sed -i -e "s/\(stroke=\"none\" fill=\"black\"\)\( font-family=\"arial\" font-size=\"4.00\"\)/\1 opacity=\"0\"\2/g" "/UKR-server-trend.svg"
and then find the data value and generate a <title>
element of that data value using back-references. I must admit, I have not memorized regular expressions to the point where I can just write these and have them work on the first try: gnu’s sed tester is very helpful.
sed -i -e "s/\(<text><tspan font-family=\"arial\" >\)\([-1234567890]*\)<\/tspan><\/text>/\1\2<title>\2<\/title><\/tspan><\/text>/g" "/UKR-server-trend.svg"
And you get hovertext data interrogation. W00t!
Note that cron
jobs are executed with different environment variables than user executed scripts, which can result in date formatting variations (which can be set explicitly in gnuplot
) and thousands separator and decimal characters (,/.). To get consistent results with a cron
job, explicitly set the appropriate locale, either in the script like
#!/bin/bash LC_NUMERIC=en_US.UTF-8 ...
or for all cron
jobs as in crontab -e
LC_NUMERIC=en_US.UTF-8 MAILTO=user@domain.com # .---------------- minute (0 - 59) # | .------------- hour (0 - 23) # | | .---------- day of month (1 - 31) # | | | .------- month (1 - 12) OR jan,feb,mar,apr ... # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat # | | | | | # * * * * * <command to be executed>
The customized SVG
file is SCP
d to the server as before, replacing the previous day’s. Repeat visitors might have to clear their cache. It’s also important to disable caching on the site for the page, for example if using wp super cache or something, because there’s no signal to the cache management engine that the file has been updated.
South Lake Tahoe Caldor Fire Timelapse
Sentinalhub Playground is an excellent resource for near real time, albeit not quite google earth 1m resolution, satellite images. One of the cool features is being able to adjust the mapping of the satellite bands to RGB outputs. For example, using Sentinel-2 L2A image data of South Lake Tahoe between 2021-08-17 and 2021-09-01 and remapping the 2190nm (SWIR2) to red, which tends to highlight fires though isn’t thermal, 783nm to green, a vegetation band (though it is NIR to humans) to make vegetation cover more obvious, and 443nm to blue instead of 490nm as shorter wavelengths tend to be scattered more by aerosols and smoke the fire line (bright red) and smoke (obvs) is very visible while vegetation is (false) green. Burnt earth shows as dark red, compared to bare ground, which tends to show tan in this mapping, thus revealing the current line of fire, the recently burned areas, and the wind direction carrying smoke, which tends to correlate with the advancing line, and fuel (vegetation) still standing.
Then using the history controller to generate and save a sequence of stills, we can animate the progress of the fire with a simple FFMPEG command:
ffmpeg -framerate 1 -pattern_type glob -i '*.jpg' -vf crop=1754:1146 -c:v libx264 -r 30 -pix_fmt yuv420p fire.mp4
and you get:
Lizard Visit
A New animal visitor came by today.
Cactus Farmer
Just a few years back, when I was in 3rd or 4th Grade, my brother and I went to visit David and Jesse Lenat at their Cactus Farm. While we were exploring the green houses, their dad, Richard, gave my brother and I each a cactus to take home.
Mine lived in a small pot near a window through the rest of grade school and high school and then my mom cared for it through college. It grew into a little cluster of pencil thin, green, spiky pads over the years.
After I graduated, moved to California and got an apartment in SF; I was home for Christmas one year and took one of the pads wrapped in tissue to California. It grew well there and now produces big, bright yellow flowers every year.
This Christmas, I stuffed two tiny buds into glass bottles and brought them to Iraq and planted in the yard with one of the cat’s help (paw in the background).
A Day Out And About In Basra
A day spent out reviewing alternate sites where unexpected underground obstructions impact construction means a chance to make new friends.
These days the attention we attract is welcome and fun.
Visiting the Burj Khalifa
Dubai is an interesting contrast to Iraq. The first time I went through DXB from BSR it was more than a little culture shock. Getting out of the airport only amplifies the experience.
Jared and I had dinner at the Mall of Dubai and before eating had a little walk around the fountains – the largest dancing fountains in the world at the foot of the tallest man-made structure in the world.
Dubai is an good place to spot cars. Obviously the gold accented rolls is more pose-worthy than the $450k GTO. Then again they were probably posing with the license plate number which I think was 1, and therefore cost as much as 20 Ferrari GTOs.