I’m extremely happy with most features of my Microsoft Surface Pro 9, but some features are annoying. One of those was what feels like the mouse cursor stuttering when I’m running on battery power. I’d also notice the animations of window resizing were not as smooth as when running on power.
As with most things, the issue with finding a solution is finding he correct terms to search. Windows 11 has a feature called Dynamic Refresh Rate. Instead of choosing 60Hz or 120Hz you can set it to dynamic and it makes it’s own choices as to how fast the refresh rate runs. I looked at the settings and mine was set to 60Hz. I changed it to Dynamic, and it at least feels responsive to me when running on battery. I’ll still have to see how much it affects the battery life.
I know that this was changing related to when I plugged in to power, but there didn’t seem to be an obvious setting under the power settings.
This is a description of how to configure NetworkManager on Raspberry Pi OS Bookworm to have stored security keys for multiple WiFi networks, and have those networks prioritized as to which is used. An example of this might be to use shared wifi from a mobile hotspot, but prefer wifi from a home gateway if it’s available. A different example is work and home networks, but those usually are not visible at the same time, so the priority is less important in that case. I’m starting using the Raspberry Pi Imager to create a fresh installation to fully understand what’s happening.
Edit the settings to assign the hostname, default username and password, and the SSID and Password for the first WiFi network you want to connect to.
It’s important to enable SSH under the services tab and include a public key to be able to connect from your desktop.
After creating the image and putting it in the Pi and booting it, the Pi should come up and connect to the specified WiFi network, at which point you can ssh into the pi and add more networks.
Here’s what’s happening behind the scenes on the first boot. The /boot/cmdline.txt file has special commands telling the system to run firstboot scripts to configure the machine.
The two interesting parts of that script are init=/usr/lib/raspberrypi-sys-mods/firstboot and systemd.run=/boot/firstrun.sh
#!/bin/bash
reboot_pi () {
umount "$FWLOC"
mount / -o remount,ro
sync
reboot -f "$BOOT_PART_NUM"
sleep 5
exit 0
}
get_variables () {
ROOT_PART_DEV=$(findmnt / -no source)
ROOT_DEV_NAME=$(lsblk -no pkname "$ROOT_PART_DEV")
ROOT_DEV="/dev/${ROOT_DEV_NAME}"
BOOT_PART_DEV=$(findmnt "$FWLOC" -no source)
BOOT_PART_NAME=$(lsblk -no kname "$BOOT_PART_DEV")
BOOT_DEV_NAME=$(lsblk -no pkname "$BOOT_PART_DEV")
BOOT_PART_NUM=$(cat "/sys/block/${BOOT_DEV_NAME}/${BOOT_PART_NAME}/partition")
OLD_DISKID=$(fdisk -l "$ROOT_DEV" | sed -n 's/Disk identifier: 0x\([^ ]*\)/\1/p')
}
fix_partuuid() {
if [ "$BOOT_PART_NUM" != "1" ]; then
return 0
fi
mount -o remount,rw "$ROOT_PART_DEV"
mount -o remount,rw "$BOOT_PART_DEV"
DISKID="$(dd if=/dev/hwrng bs=4 count=1 status=none | od -An -tx4 | cut -c2-9)"
fdisk "$ROOT_DEV" > /dev/null <<EOF
x
i
0x$DISKID
r
w
EOF
if [ "$?" -eq 0 ]; then
sed -i "s/${OLD_DISKID}/${DISKID}/g" /etc/fstab
sed -i "s/${OLD_DISKID}/${DISKID}/" "$FWLOC/cmdline.txt"
sync
fi
mount -o remount,ro "$ROOT_PART_DEV"
mount -o remount,ro "$BOOT_PART_DEV"
}
regenerate_ssh_host_keys () {
mount -o remount,rw /
/usr/lib/raspberrypi-sys-mods/regenerate_ssh_host_keys
RET="$?"
mount -o remount,ro /
return "$RET"
}
apply_custom () {
CONFIG_FILE="$1"
mount -o remount,rw /
mount -o remount,rw "$FWLOC"
if ! python3 -c "import toml" 2> /dev/null; then
FAIL_REASON="custom.toml provided, but python3-toml is not installed\n$FAIL_REASON"
else
set -o pipefail
/usr/lib/raspberrypi-sys-mods/init_config "$CONFIG_FILE" |& tee /run/firstboot.log | while read -r line; do
MSG="$MSG\n$line"
whiptail --infobox "$MSG" 20 60
done
if [ "$?" -ne 0 ]; then
mv /run/firstboot.log /var/log/firstboot.log
FAIL_REASON="Failed to apply customisations from custom.toml\n\nLog file saved as /var/log/firstboot.log\n$FAIL_REASON"
fi
set +o pipefail
fi
rm -f "$CONFIG_FILE"
mount -o remount,ro "$FWLOC"
mount -o remount,ro /
}
main () {
get_variables
whiptail --infobox "Generating SSH keys..." 20 60
regenerate_ssh_host_keys
if [ -f "$FWLOC/custom.toml" ]; then
MSG="Applying customisations from custom.toml...\n"
whiptail --infobox "$MSG" 20 60
apply_custom "$FWLOC/custom.toml"
fi
whiptail --infobox "Fix PARTUUID..." 20 60
fix_partuuid
return 0
}
mountpoint -q /proc || mount -t proc proc /proc
mountpoint -q /sys || mount -t sysfs sys /sys
mountpoint -q /run || mount -t tmpfs tmp /run
mkdir -p /run/systemd
mount / -o remount,ro
if ! FWLOC=$(/usr/lib/raspberrypi-sys-mods/get_fw_loc); then
whiptail --msgbox "Could not determine firmware partition" 20 60
poweroff -f
fi
mount "$FWLOC" -o rw
sed -i 's| init=/usr/lib/raspberrypi-sys-mods/firstboot||' "$FWLOC/cmdline.txt"
sed -i 's| sdhci\.debug_quirks2=4||' "$FWLOC/cmdline.txt"
if ! grep -q splash "$FWLOC/cmdline.txt"; then
sed -i "s/ quiet//g" "$FWLOC/cmdline.txt"
fi
mount "$FWLOC" -o remount,ro
sync
main
if [ -z "$FAIL_REASON" ]; then
whiptail --infobox "Rebooting in 5 seconds..." 20 60
sleep 5
else
whiptail --msgbox "Failed running firstboot:\n${FAIL_REASON}" 20 60
fi
reboot_pi
/boot/firstrun.sh
#!/bin/bash
set +e
CURRENT_HOSTNAME=`cat /etc/hostname | tr -d " \t\n\r"`
if [ -f /usr/lib/raspberrypi-sys-mods/imager_custom ]; then
/usr/lib/raspberrypi-sys-mods/imager_custom set_hostname WimPi4-Dev
else
echo WimPi4-Dev >/etc/hostname
sed -i "s/127.0.1.1.*$CURRENT_HOSTNAME/127.0.1.1\tWimPi4-Dev/g" /etc/hosts
fi
FIRSTUSER=`getent passwd 1000 | cut -d: -f1`
FIRSTUSERHOME=`getent passwd 1000 | cut -d: -f6`
if [ -f /usr/lib/raspberrypi-sys-mods/imager_custom ]; then
/usr/lib/raspberrypi-sys-mods/imager_custom enable_ssh -k 'ssh-rsa MyReallyLongPublicKeyWasHere wim@MyDesktop'
else
install -o "$FIRSTUSER" -m 700 -d "$FIRSTUSERHOME/.ssh"
install -o "$FIRSTUSER" -m 600 <(printf "ssh-rsa MyReallyLongPublicKeyWasHere wim@MyDesktop") "$FIRSTUSERHOME/.ssh/authorized_keys"
echo 'PasswordAuthentication no' >>/etc/ssh/sshd_config
systemctl enable ssh
fi
if [ -f /usr/lib/userconf-pi/userconf ]; then
/usr/lib/userconf-pi/userconf 'wim' 'MyHashedPasswordWasHere'
else
echo "$FIRSTUSER:"'MyHashedPasswordWasHere' | chpasswd -e
if [ "$FIRSTUSER" != "wim" ]; then
usermod -l "wim" "$FIRSTUSER"
usermod -m -d "/home/wim" "wim"
groupmod -n "wim" "$FIRSTUSER"
if grep -q "^autologin-user=" /etc/lightdm/lightdm.conf ; then
sed /etc/lightdm/lightdm.conf -i -e "s/^autologin-user=.*/autologin-user=wim/"
fi
if [ -f /etc/systemd/system/getty@tty1.service.d/autologin.conf ]; then
sed /etc/systemd/system/getty@tty1.service.d/autologin.conf -i -e "s/$FIRSTUSER/wim/"
fi
if [ -f /etc/sudoers.d/010_pi-nopasswd ]; then
sed -i "s/^$FIRSTUSER /wim /" /etc/sudoers.d/010_pi-nopasswd
fi
fi
fi
if [ -f /usr/lib/raspberrypi-sys-mods/imager_custom ]; then
/usr/lib/raspberrypi-sys-mods/imager_custom set_wlan 'Sola' 'MyHashedWiFiPasswordWasHere' 'US'
else
cat >/etc/wpa_supplicant/wpa_supplicant.conf <<'WPAEOF'
country=US
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
ap_scan=1
update_config=1
network={
ssid="Sola"
psk=MyHashedWiFiPasswordWasHere
}
WPAEOF
chmod 600 /etc/wpa_supplicant/wpa_supplicant.conf
rfkill unblock wifi
for filename in /var/lib/systemd/rfkill/*:wlan ; do
echo 0 > $filename
done
fi
if [ -f /usr/lib/raspberrypi-sys-mods/imager_custom ]; then
/usr/lib/raspberrypi-sys-mods/imager_custom set_keymap 'us'
/usr/lib/raspberrypi-sys-mods/imager_custom set_timezone 'America/Los_Angeles'
else
rm -f /etc/localtime
echo "America/Los_Angeles" >/etc/timezone
dpkg-reconfigure -f noninteractive tzdata
cat >/etc/default/keyboard <<'KBEOF'
XKBMODEL="pc105"
XKBLAYOUT="us"
XKBVARIANT=""
XKBOPTIONS=""
KBEOF
dpkg-reconfigure -f noninteractive keyboard-configuration
fi
rm -f /boot/firstrun.sh
sed -i 's| systemd.run.*||g' /boot/cmdline.txt
exit 0
The first of those two files is standard to all Raspberry Pi first boots, the second has the personalized data for the system. /boot/firstrun.sh uses the /usr/lib/raspberrypi-sys-mods/imager_custom command on line 42 to set the WiFi ssid and password. In OS releases Bullseye and prior the WiFi was managed via the wpa_supplicant.conf file, while in Bookworm the WiFi is managed by NetworkManager.
#!/bin/sh
set -e
usage () {
if [ "$#" -eq "0" ]; then
usage set_hostname enable_ssh set_wlan set_keymap set_timezone
return 0
fi
echo "Usage: "
for arg in "$@"; do
case "$arg" in
set_hostname)
echo " $0 set_hostname HOSTNAME"
;;
import_ssh_id)
echo " $0 import_ssh_id USERID1 [USERID2]..."
;;
enable_ssh)
echo " $0 enable_ssh [-k|--key-only]|[-p|--pass-auth] [-d|--disabled] [KEY_LINE1 [KEY_LINE2]...]"
;;
set_wlan)
echo " $0 set_wlan [-h|--hidden] [-p|--plain] SSID [PASS [COUNTRY]]"
;;
set_wlan_country)
echo " $0 set_wlan_country COUNTRY]"
;;
set_keymap)
echo " $0 set_keymap KEYMAP"
;;
set_timezone)
echo " $0 set_timezone TIMEZONE"
;;
esac
done
}
set_hostname () (
if [ "$#" -ne 1 ]; then
usage set_hostname
exit 1
fi
HOSTNAME="$1"
raspi-config nonint do_hostname "$HOSTNAME"
echo "$HOSTNAME" > /etc/hostname
)
FIRSTUSER=$(getent passwd 1000 | cut -d: -f1)
FIRSTUSERHOME=$(getent passwd 1000 | cut -d: -f6)
SSH_DIR="$FIRSTUSERHOME/.ssh"
AUTHORISED_KEYS_FILE="$SSH_DIR/authorized_keys"
add_ssh_keys () (
if ! [ -d "$SSH_DIR" ]; then
install -o "$FIRSTUSER" -g "$FIRSTUSER" -m 700 -d "$SSH_DIR"
fi
for key in "$@"; do
echo "$key" >> "$AUTHORISED_KEYS_FILE"
done
if [ -f "$AUTHORISED_KEYS_FILE" ]; then
chmod 600 "$AUTHORISED_KEYS_FILE"
chown "$FIRSTUSER:$FIRSTUSER" "$AUTHORISED_KEYS_FILE"
fi
)
enable_ssh () (
ENABLE=1
KEY_ONLY_SED_STR='s/^[#\s]*PasswordAuthentication\s\+\S\+$/PasswordAuthentication no/'
PASSAUTH_SED_STR='s/^[#\s]*PasswordAuthentication\s\+\S\+$/PasswordAuthentication yes/'
for arg in "$@"; do
if [ "$arg" = "-k" ] || [ "$arg" = "--key-only" ]; then
sed -i "$KEY_ONLY_SED_STR" /etc/ssh/sshd_config
elif [ "$arg" = "-p" ] || [ "$arg" = "--pass-auth" ]; then
sed -i "$PASSAUTH_SED_STR" /etc/ssh/sshd_config
elif [ "$arg" = "-d" ] || [ "$arg" = "--disabled" ]; then
ENABLE=0
else
add_ssh_keys "$arg"
fi
done
if [ "$ENABLE" = 1 ]; then
systemctl -q enable ssh
fi
)
set_wlan_country () (
if [ "$#" -ne 1 ]; then
usage set_wlan_country
exit 1
fi
# shellcheck disable=SC2030
COUNTRY="$1"
raspi-config nonint do_wifi_country "$COUNTRY"
)
set_wlan () (
HIDDEN="false"
PLAIN=0
for arg in "$@"; do
# shellcheck disable=SC2031
if [ "$arg" = "-h" ] || [ "$arg" = "--hidden" ]; then
HIDDEN="true"
elif [ "$arg" = "-p" ] || [ "$arg" = "--plain" ]; then
PLAIN=1
elif [ -z "${SSID+set}" ]; then
SSID="$arg"
elif [ -z "${PASS+set}" ]; then
PASS="$arg"
elif [ -z "${COUNTRY+set}" ]; then
COUNTRY="$arg"
else
usage set_wlan
exit 1
fi
done
if [ -z "${SSID+set}" ]; then
usage set_wlan
exit 1
fi
if [ -n "$COUNTRY" ]; then
set_wlan_country "$COUNTRY"
fi
CONNFILE=/etc/NetworkManager/system-connections/preconfigured.nmconnection
UUID=$(uuid -v4)
cat <<- EOF >${CONNFILE}
[connection]
id=preconfigured
uuid=${UUID}
type=wifi
[wifi]
mode=infrastructure
ssid=${SSID}
hidden=${HIDDEN}
[ipv4]
method=auto
[ipv6]
addr-gen-mode=default
method=auto
[proxy]
EOF
if [ ! -z "${PASS}" ]; then
cat <<- EOF >>${CONNFILE}
[wifi-security]
key-mgmt=wpa-psk
psk=${PASS}
EOF
fi
# NetworkManager will ignore nmconnection files with incorrect permissions,
# to prevent Wi-Fi credentials accidentally being world-readable.
chmod 600 ${CONNFILE}
)
import_ssh_id () (
SCRIPT='/var/lib/raspberrypi-sys-mods/import-ssh'
if [ "$#" -eq 0 ]; then
usage import_ssh_id
exit 1
fi
if ! command -v ssh-import-id > /dev/null; then
echo "ssh-import-id not available"
exit 1
fi
mkdir -p "$(dirname "$SCRIPT")"
# shellcheck disable=SC2094
cat <<- EOF > "$SCRIPT"
#!/bin/sh
COUNTER=0
while [ "\$COUNTER" -lt 10 ]; do
COUNTER=\$((COUNTER + 1))
if runuser -u \$(getent passwd 1000 | cut -d: -f1) -- ssh-import-id $@; then
break
fi
sleep 5
done
systemctl stop import-ssh.timer
systemctl disable import-ssh.timer
rm -f "\$0"
rm -f /etc/systemd/system/import-ssh.timer
rm -f /etc/systemd/system/import-ssh.service
rmdir --ignore-fail-on-non-empty "$(dirname "$SCRIPT")"
EOF
chmod 700 "$SCRIPT"
cat <<- EOF > /etc/systemd/system/import-ssh.timer
[Unit]
Description=Import SSH keys using ssh-import-id
[Timer]
OnBootSec=1
OnUnitActiveSec=10
[Install]
WantedBy=timers.target
EOF
cat <<- EOF > /etc/systemd/system/import-ssh.service
[Unit]
Description=Import SSH keys using ssh-import-id
After=network-online.target userconfig.service
[Service]
Type=oneshot
ExecStart=$SCRIPT
EOF
ln -f -s /etc/systemd/system/import-ssh.timer \
/etc/systemd/system/timers.target.wants/import-ssh.timer
)
set_keymap () (
if [ "$#" -ne 1 ]; then
usage set_keymap
exit 1
fi
raspi-config nonint do_configure_keyboard "$1"
)
set_timezone () (
if [ "$#" -ne 1 ]; then
usage set_timezone
exit 1
fi
raspi-config nonint do_change_timezone "$1"
)
if [ "$#" -eq 0 ]; then
echo "No command specified"
usage
exit 1
fi
command="$1"; shift
case "$command" in
set_hostname|import_ssh_id|enable_ssh|set_wlan_country|set_wlan|set_keymap|set_timezone)
"$command" "$@"
;;
*)
echo "Unsupported command: $command"
usage
exit 1
;;
esac
The interesting lines in that script are from 97 to 156 in the set_wlan() function. It directly creates a file named /etc/NetworkManager/system-connections/preconfigured.nmconnection with some specific sections. It’s too bad that the script doesn’t currently invoke the nmcli command that I’ll be using to create the secondary networks and modify the original network because the underlying filename appears to be fixed once it’s created and it would be nice if it matched the wifi network name instead of being listed as preconfigured.
I’m finally getting to the subject of this post. the nmcli (Network Manager Command Line) program is what I’ll be using to modify the network settings. One of the nice features of nmcli is that it works well with using the tab key for command completion. Many of the parameters can be abbreviated, but I’ll be hitting tab to have the full length versions displayed. In the previous image you can see the command I used (nmcli device wifi list) to list the available WiFi networks and the command to list the configured connections (nmcli connection show). I also listed the directory that NetworkManager stores its connection data to see the file that was created by the firstrun script. The asterisk to the left of Sola shows that I’m connected to the 5GHz (channel 40) version of my local network. The colors give a nice indication of signal strength.
Before I make any changes, the contents of the preconfigured file are:
First I’m going to change the id from preconfigured to a descriptive network name so that when I list the connections it it more useful: sudo nmcli connection modify preconfigured connection.id wifi-sola connection.autoconnect-priority 4
You can see that it’s changed the contents of the file, but not changed the filename itself. The new contents are:
I went ahead and modified the connection priority with the same command. I then followed it up with a command to create a connection to my other WiFi network: sudo nmcli connection add con-name wifi-wimsworld type wifi ssid "WimsWorld" wifi-sec.key-mgmt wpa-psk wifi-sec.psk "MyWiFiPassword" connection.autoconnect-priority 5
The larger priority of 5 specified for WimsWorld tells network manager to prefer it over Sola with a priority of 4. I rebooted the machine and when it came up it was connected to WimsWorld instead of Sola. (I also didn’t know the IP address and because it was now on a different network segment mDNS name resolution no longer works, so I plugged an ethernet cable into the device so I could easily find it.)
I recently added AIS Dispatcher to the software running on my raspberry pi on my boat. The Pi is running connected to my NMEA 2000 (N2K) network and receives AIS messages into SignalK via that path, but I chose to configure AIS Dispatcher to communicate with my Emtrak B954 AIS Transceiver directly over the local network and WiFi. When I set up my emtrak I’d configured it so that it uses the WiFi as a client into my existing network and configured my hotspot to always give the emtrak the same IP address.
The installer wanted the package aha to be installed to provide context coloring on the terminal.
wim@WimPi4-Sola:~ $ sudo apt install aha
wim@WimPi4-Sola:~ $ apt info aha
Package: aha
Version: 0.5.1-3
Priority: optional
Section: utils
Maintainer: Axel Beckert <abe@debian.org>
Installed-Size: 50.2 kB
Depends: libc6 (>= 2.17)
Homepage: https://github.com/theZiz/aha
Tag: implemented-in::c, interface::commandline, role::program,
scope::utility, use::converting, works-with-format::TODO,
works-with-format::html, works-with::TODO, works-with::text
Download-Size: 16.9 kB
APT-Manual-Installed: yes
APT-Sources: http://deb.debian.org/debian bookworm/main arm64 Packages
Description: ANSI color to HTML converter
aha (ANSI HTML Adapter) converts ANSI colors to HTML, e.g. if you
want to publish the output of ls --color=yes, git diff, ccal or htop
as static HTML somewhere.
I ran the install code from the AIS Dispatcher website:
It’s important that I modified the /home/ais/etc/aiscontrol.cfg file to change the listen_port to a port other than 8080 because I’ve already got something responding on that port. I arbitrarily used 8090.
After logging into the service on my local machine I configured it to retrieve data from the emtrak on it’s local address and port. 192.168.50.23 and 5000.
After getting AIS Dispatcher up and running I went to https://www.marinetraffic.com/en/users/my_account/stations/index and added a station. Marine Traffic sent me a custom IP address and port to submit data via email. I entered that data on the Host 2 line of the AIS Dispatcher interface and restarted the service.
My main reason for doing this was to upgrade my MarineTraffic and VesselTracker accounts from the free account to have a few more features. I do a similar thing submitting data to both FlightAware and FlightRadar24. I also like contributing the the general recorded data when it’s easy to do so.
After my complaint about Apple Music and Apple Devices, I decided to attempt to go back to iTunes. At least that appears to have worked the way I hoped. I uninstalled both of the new apps and rebooted my machine then loaded iTunes. It still saw all of my music including the album artwork including the play count metadata from before the switch to Apple Music.
The music play counts that I’d progressed by playing music using the Apple music App seem to have been lost, but I’m much happier with the old interface.
I also like the more dense data delivery with smaller text and less unnecessary white space. It’s funny to recognize that it was already eleven years ago that I was complaining about information density on iOS7 and now we’re on iOS17.
I have complained for years about the quality of Apple software and their recent upgrade from iTunes to separate Apple Devices and Apple Music software is no exception. by installing these two pieces of software from the Microsoft Store, the already installed iTunes was degraded to only support Podcasts and Audio Books.
I wouldn’t mind this if the Music App functioned properly and hadn’t already lost album artwork on close to 50% of the albums in my library.
In the past I’d been able to right click on albums and tell it to look for missing artwork, but that option isn’t available to me in the current app. I’ve also been able to go into preferences and specify details about how I want to import and manage my music. I’ve been importing music in Apple Lossless Codec directly from CDs. None of those options are visible under the settings, even when it appears they might be under the advanced button.
What I find just as frustrating as the missing artwork for so many of my albums is how unresponsive the software is. It’s obviously using some display framework that has added more layers of code between the user interface and the underlying data and I’m often staring at a blank screen with a spinning status indicator while it decides what it’s going to show.
Trying the Apple Devices app to backup my iPhone to my computer, I was at least happy to see that it recognized the last time I’d backed up my iPhone, but it wasn’t listing the music, saying that it couldn’t open the Apple Music App, even though I’ve got the music app opened.
After it backed up my phone, it gave some message about some of the music files not being able to be backed up.
While this is a good sign, I can’t help but wonder what may not be properly backed up or synchronized.  In the past I’ve always liked that iTunes would keep track of when I last played a particular song, as well as how many times the song had been played. This data would be accumulated and synchronized with my computer and iPhone. It’s one more reason I was sucked into the Apple iPhone ecosystem.
I was pleased that The backup process didn’t cause OneDrive to start downloading my photos to my local machine.
Using iTunes, I’d had a similar setting during backup of my iPhone, that Photos are not to be synced to my computer. I have OneDrive running on my iPhone and it backs up my photos directly to the cloud. With ITunes the backup process would still try to access my local Pictures directory even though I had synchronization turned off. Each time I backed up my phone, I’d have to enable sync, switch the directory to a non-ondedrive-mirrored directory, then disable sync again. That way even if iTunes looked at the contents of every file in the specified directory it wasn’t trying to retrieve files from the cloud.
In a previous post I talked about learning mDNS programming in windows to make my program WimTiVoServer work with my new TiVo Bolt.
I spent much longer than necessary trying to get this to work because the TiVo rejects devices the declare the TXT record TSN= while it will accept them with the TXT record tsn=. Evidently the uppercase version is reserved for official TiVo devices and the TiVo will attempt to check the ID with their account management system.
What follows is my working code to register the software in mDNS.
// My callback function https://learn.microsoft.com/en-us/windows/win32/api/windns/nc-windns-dns_service_register_complete
void DnsServiceRegisterComplete(DWORD Status, PVOID pQueryContext,PDNS_SERVICE_INSTANCE pInstance)
{
if (bConsoleExists)
{
std::wcout << L"[" << getwTimeISO8601() << L"] DnsServiceRegisterComplete(" << Status << L") DNS_ERROR_RCODE_SERVER_FAILURE=" << DNS_ERROR_RCODE_SERVER_FAILURE << std::endl;
if (0 == Status)
{
std::wcout << L"[ ] pInstance->pszInstanceName " << pInstance->pszInstanceName << std::endl;
std::wcout << L"[ ] pInstance->pszHostName " << pInstance->pszHostName << std::endl;
std::wcout << L"[ ] pInstance->wPort " << pInstance->wPort << std::endl;
if (pInstance->ip4Address != 0)
{
in_addr ipAddr;
ipAddr.S_un.S_addr = *pInstance->ip4Address;
std::wcout << L"[ ] pInstance->ip4Address " << inet_ntoa(ipAddr) << std::endl;
}
auto index = pInstance->dwPropertyCount;
while (index-- > 0)
std::wcout << L"[ ] pInstance->keys[" << index << L"]=pInstance->values[" << index << L"] " << pInstance->keys[index] << L"=" << pInstance->values[index] << std::endl;
}
}
return;
}
void TiVomDNSRegister(bool enable = true)
{
// The following link is an intersting book related to ZeroConf.
// https://flylib.com/books/en/2.94.1.53/1/
if (ControlSocket != INVALID_SOCKET)
{
struct sockaddr addr;
addr.sa_family = AF_UNSPEC;
socklen_t addr_len = sizeof(addr);
getsockname(ControlSocket, &addr, &addr_len);
if (addr.sa_family == AF_INET)
{
struct sockaddr_in* saServer = (sockaddr_in*)&addr;
WCHAR szHostName[256] = TEXT("");
DWORD dwSize = sizeof(szHostName);
GetComputerNameEx(ComputerNameDnsHostname, szHostName, &dwSize);
// Here I'm creating and destroying an instance just to test the function call.
std::wstring MyServiceName(L"._tivo-videos._tcp.local"); MyServiceName.insert(0, szHostName);
std::wstring MyHostName(L".local"); MyHostName.insert(0, szHostName);
std::wstring MyPath(L"/TiVoConnect?Command=QueryContainer\&Container="); MyPath.append(szHostName);
std::wstring MyPlatform(std::wstring_convert<std::codecvt_utf8<wchar_t>>().from_bytes(myServer.m_platform));
std::wstring MyVersion(std::wstring_convert<std::codecvt_utf8<wchar_t>>().from_bytes(myServer.m_swversion));
std::wstring MyID(std::wstring_convert<std::codecvt_utf8<wchar_t>>().from_bytes(myServer.m_identity));
// These Keys are copied from what a real TiVo registers. These Values should be set based on the same Items I'm putting in the beacon
std::vector<PCWSTR> keys;
std::vector<PCWSTR> values;
keys.push_back(L"protocol"); values.push_back(L"http");
keys.push_back(L"path"); values.push_back(MyPath.c_str());
keys.push_back(L"swversion"); values.push_back(MyVersion.c_str());
keys.push_back(L"platform"); values.push_back(MyPlatform.c_str());
keys.push_back(L"tsn"); values.push_back(MyID.c_str()); // 2024-01-24 If this is not lowercase the TiVo will reject the device as not in your TiVo account. (tivo.com/help/SH06)
auto MyServiceInstancePtr = DnsServiceConstructInstance(
MyServiceName.c_str(),
MyHostName.c_str(),
nullptr,
nullptr,
ntohs(saServer->sin_port),
0,
0,
keys.size(),
&keys[0],
&values[0]
);
DNS_SERVICE_REGISTER_REQUEST rd = { 0 };
rd.Version = DNS_QUERY_REQUEST_VERSION1;
rd.InterfaceIndex = 0;
rd.pServiceInstance = MyServiceInstancePtr;
rd.pRegisterCompletionCallback = &DnsServiceRegisterComplete;
rd.pQueryContext = nullptr;
rd.hCredentials = 0;
rd.unicastEnabled = FALSE; // true if the DNS protocol should be used to advertise the service; false if the mDNS protocol should be used.
if (enable)
{
auto mDNSReturn = DnsServiceRegister(&rd, nullptr);
if (bConsoleExists)
std::cout << "[" << getTimeISO8601() << "] DnsServiceRegister(" << mDNSReturn << ") DNS_REQUEST_PENDING=" << DNS_REQUEST_PENDING << std::endl;
}
else
{
auto mDNSReturn = DnsServiceDeRegister(&rd, nullptr);
if (bConsoleExists)
std::cout << "[" << getTimeISO8601() << "] DnsServiceRegister(" << mDNSReturn << ") DNS_REQUEST_PENDING=" << DNS_REQUEST_PENDING << std::endl;
}
DnsServiceFreeInstance(MyServiceInstancePtr);
}
}
return;
}
I’d had issues setting up cross platform development in the past and written about it in 2022. Today I was fighting a similar problem while getting the system working with the latest version, Bookworm, of Pi Operating System. I was able to ssh from a command line to the account I wanted to use and specify the identity file and everything worked fine. When I tried from either the dialog in visual studio or the command line tool it failed.
I came across an obscure post saying that adding PubkeyAcceptedAlgorithms +ssh-rsa to /etc/ssh/sshd_config and restarting sshd would solve the problem. Because the newer system includes files in /etc/ssh/sshd_config.d/ I created a file /etc/ssh/sshd_config.d/visualstudio.conf with the single line PubkeyAcceptedAlgorithms +ssh-rsa and restarted sshd and was able to get the command line on my windows machine to add the connection.
On my development Pi I set up a dedicated user, visualstudio, that’s part of the same group my primary user, and then I allow group access the visualstudio home directory.
I wanted to update a raspberry pi system from bullseye to bookworm and thought the best backup would be to shutdown the system, remove the memory card, and fully back it up in another machine.
Unfortunately getting the sd card out of the pi in its case was difficult and I attempted to grab it with a pair of tweezers. The scratches on the back of this card appear to go through the very thin plastic and damage the electronics.
When I loaded the drive in another computer it only saw a /dev/sdb volume, and not the /dev/sdb1 and /dev/sdb2 that I was hoping. When I tried putting it back into the original Pi, it never booted. Very Frustrating lesson to learn as I lost data that had been accumulating on this device for several years.
A friend bought a new Govee H5105 Thermometer and since they were on sale at Amazon I picked one up to see if my software could record it’s data or needed changes to support it. The image makes the thermometer look huge next to a phone. It is smaller than any of the other devices with displays. Its display is an e-Ink technology with a shiny plastic layer over the top. The amazon details claim temperature accuracy to ±0.54℉/±0.3℃, and humidity accuracy at ±3%RH. The older devices I’ve been using have been ±0.2℃ which indicates a new sensor is being used. This uses a CR2450 button type battery with claims to up to a year. I’ll be interested to see how long it lasts.
I bought a Govee H5100 device last August and it’s been supported by my data recording software since then. The H5100 uses a single AAA battery and has no display. It also is described using the same temperature and humidity accuracy, ±0.54℉/±0.3℃ and ±3%RH, as the h5105. The h5100 battery died on January 12th, just under five months. I replaced the battery and will see if the first time frame is what to expect from the H5100 device, or just a bad starting battery.
Each of these devices broadcasts temperature data using BLE (Bluetooth Low Energy) advertisements. The advertising data is similar to the other Govee devices I’ve used, so supporting them was extremely easy. Unfortunately, I have not managed to retrieve any stored data from the h5100 in the time it’s been in use, leading me to believe that it’s either a different connected protocol or at least a different GUID from the older Govee devices. The fact that my software has not been able to download data from the h5100 may have contributed to the speedy battery usage because it attempts to make a connection to the device and download data and will keep retrying when data hasn’t been downloaded in the past two weeks. I believe that a BLE connection requires significantly more power than a BLE advertisement.
At some point I’ll have to dig out my old Android Tablet, enable developer mode and Bluetooth logging, run the Govee App, and get a capture of the data transferred to understand what’s required to receive historical data from the device. The very nature of doing that is a tedious set of steps and I was just searching to see if I’d documented it but couldn’t find mention of it here.
I wanted to upgrade my TiVo Roamio to a TiVo Bolt because the Bolt has a slightly newer processor and supports streaming videos from itself to the iPhone App. I bought a Bolt on Ebay and added it to my account. It can see the Roamio, but didn’t see my software that I use to transfer videos from my computer to the TiVo.
My software was written a long time ago and advertised itself using UDP beacons. I’d originally written my software because the TiVo Desktop software was frustrating to maintain. It wanted to install Apple Bonjour Services, which then conflicted with the installation of Bonjour services included in Apple iTunes. Neither installer did a good job of recognizing or negotiating to use the most recent version of Bonjour. My software is small, only relies on FFMPEG being in the system path, and can run either as a windows service or on the command line. FFMPEG supports many more input formats than the TiVo Desktop software did, is constantly being developed, and I can update it as frequently as I like.
With the Bolt, I finally understood the reason TiVo had installed Bonjour was that TiVo had migrated to preferring using ZeroConf for service discovery on the local network instead of UDP. I’ve also learned that Bonjour is simply Apple’s service name supporting ZeroConf. ZeroConf is predominantly built on Multicast DNS, or mDNS. Bonjour had earlier been known as Rendezvous.
The Windows API supports now mDNS, since approximately version 10.0.18362.0 (1903/19H1, May 2019.) I should be able to add mDNS advertising to my program, but finding examples has been the hard part. I’m supplying my current code here in hopes that it helps someone in the future.
I added some code to handle browsing for the name _tivo-videos._tcp.local to my program that listens for the UDP beacons. The picture at the top shows my UDP Beacon Listener running after I’ve enabled my browse function with the query for _tivo-videos._tcp.local entries.
The TiVo itself advertises the following services:
_http._tcp.local
_tivo-remote._tcp.local
_tivo-device._tcp.local
_tivo-videos._tcp.local
_tivo-videostream._tcp.local
_tivo-mindrpc._tcp.local
// https://stackoverflow.com/questions/66474722/use-multicast-dns-when-network-cable-is-unplugged
// https://git.walbeck.it/archive/mumble-voip_mumble/src/branch/master/src/mumble/Zeroconf.cpp
DNS_SERVICE_CANCEL cancelBrowse{ 0 }; // A pointer to a DNS_SERVICE_CANCEL structure that can be used to cancel a pending asynchronous browsing operation. This handle must remain valid until the query is canceled.
VOID WINAPI Browse_mDNS_Callback(DWORD Status, PVOID pQueryContext, PDNS_RECORD pDnsRecord)
{
if (Status != ERROR_SUCCESS)
{
if (pDnsRecord)
DnsRecordListFree(pDnsRecord, DnsFreeRecordList);
if (Status == ERROR_CANCELLED)
std::wcout << "[ ] DnsServiceBrowse() reports status code ERROR_CANCELLED" << std::endl;
else
std::wcout << "[ ] DnsServiceBrowse() reports status code " << Status << ", ignoring results" << std::endl;
}
if (!pDnsRecord)
return;
std::cout << "[" << getTimeISO8601() << "] Browse_mDNS_Callback" << std::endl;
for (auto cur = pDnsRecord; cur; cur = cur->pNext)
{
std::wcout << "[ ] pName: " << std::wstring(cur->pName) << std::endl;
switch (cur->wType)
{
case DNS_TYPE_PTR:
std::wcout << "[ ] Data.PTR.pNameHost: " << std::wstring(cur->Data.PTR.pNameHost) << std::endl;
break;
case DNS_TYPE_A: // RFC 1034/1035
{
wchar_t StringBuf[17];
std::wcout << "[ ] Data.A.IpAddress: " << InetNtopW(AF_INET, &(cur->Data.A.IpAddress), StringBuf, sizeof(StringBuf)) << std::endl;
}
break;
case DNS_TYPE_AAAA:
{
wchar_t StringBuf[47];
std::wcout << "[ ] Data.A.IpAddress: " << InetNtopW(AF_INET6, &(cur->Data.A.IpAddress), StringBuf, sizeof(StringBuf)) << std::endl;
}
break;
case DNS_TYPE_TEXT: // RFC 1034/1035
{
auto index = cur->Data.TXT.dwStringCount;
while (index > 0)
{
index--;
std::wcout << "[ ] Data.TXT.pStringArray[" << index <<"]: " << std::wstring(cur->Data.TXT.pStringArray[index]) << std::endl;
}
}
break;
case DNS_TYPE_SRV: // RFC 2052 (Service location)
std::wcout << "[ ] Data.SRV.pNameTarget: " << std::wstring(cur->Data.SRV.pNameTarget) << std::endl;
std::wcout << "[ ] Data.SRV.wPort: " << cur->Data.SRV.wPort << std::endl;
break;
default:
std::wcout << "[ ] wType: " << cur->wType << std::endl;
}
}
DnsRecordListFree(pDnsRecord, DnsFreeRecordList);
}
void Browse_mDNS(void)
{
DNS_SERVICE_BROWSE_REQUEST browseRequest{0};
browseRequest.Version = DNS_QUERY_REQUEST_VERSION1;
browseRequest.InterfaceIndex = 0;
browseRequest.QueryName = L"_tivo-videos._tcp.local";
browseRequest.pBrowseCallback = Browse_mDNS_Callback;
browseRequest.pQueryContext = (PVOID)43;
const DNS_STATUS result = DnsServiceBrowse(&browseRequest, &cancelBrowse);
if (result == DNS_REQUEST_PENDING)
std::cout << "[" << getTimeISO8601() << "] DnsServiceBrowse(DNS_REQUEST_PENDING)" << std::endl;
else
std::cout << "[" << getTimeISO8601() << "] DnsServiceBrowse(" << result << ") DNS_REQUEST_PENDING=" << DNS_REQUEST_PENDING << std::endl;
}
void Browse_mDNS_Cancel(void)
{
auto result = DnsServiceBrowseCancel(&cancelBrowse);
if (result != ERROR_SUCCESS)
std::wcout << "[ ] DnsServiceBrowseCancel(" << result << ")" << std::endl;
}
[/code]
I call the function Browse_mDNS() and it starts looking for the entries I’m interested in. I wrote the browsing code after I had already added code to publish my details because I wanted a consistent set of results to compare my output to the TiVo output and other software that should work with the TiVo.
Below is my code that registers my service with mDNS. I’m only registering under one name, _tivo-videos._tcp.local. The Roamio and Bolt each register multiple services as they have more features to communicate with each other, but I’m only interested in providing the simple transfer of videos from my pc to the TiVo.
// My callback function https://learn.microsoft.com/en-us/windows/win32/api/windns/nc-windns-dns_service_register_complete
void DnsServiceRegisterComplete(DWORD Status, PVOID pQueryContext,PDNS_SERVICE_INSTANCE pInstance)
{
if (bConsoleExists)
{
std::wcout << L"[" << getwTimeISO8601() << L"] DnsServiceRegisterComplete(" << Status << L") DNS_ERROR_RCODE_SERVER_FAILURE=" << DNS_ERROR_RCODE_SERVER_FAILURE << std::endl;
if (0 == Status)
{
std::wcout << L"[ ] pInstance->pszInstanceName " << pInstance->pszInstanceName << std::endl;
std::wcout << L"[ ] pInstance->pszHostName " << pInstance->pszHostName << std::endl;
std::wcout << L"[ ] pInstance->wPort " << pInstance->wPort << std::endl;
if (pInstance->ip4Address != 0)
{
in_addr ipAddr;
ipAddr.S_un.S_addr = *pInstance->ip4Address;
std::wcout << L"[ ] pInstance->ip4Address " << inet_ntoa(ipAddr) << std::endl;
}
auto index = pInstance->dwPropertyCount;
while (index-- > 0)
std::wcout << L"[ ] pInstance->keys[" << index << L"]:pInstance->values[" << index << L"] " << pInstance->keys[index] << L"=" << pInstance->values[index] << std::endl;
}
}
return;
}
void TiVomDNSRegister(bool enable = true)
{
if (ControlSocket != INVALID_SOCKET)
{
struct sockaddr addr;
addr.sa_family = AF_UNSPEC;
socklen_t addr_len = sizeof(addr);
getsockname(ControlSocket, &addr, &addr_len);
if (addr.sa_family == AF_INET)
{
struct sockaddr_in* saServer = (sockaddr_in*)&addr;
WCHAR szHostName[256] = TEXT("");
DWORD dwSize = sizeof(szHostName);
GetComputerNameEx(ComputerNameDnsHostname, szHostName, &dwSize);
// Here I'm creating and destroying an instance just to test the function call.
std::wstring MyServiceName(L"._tivo-videos._tcp.local"); MyServiceName.insert(0, szHostName);
std::wstring MyHostName(L".local"); MyHostName.insert(0, szHostName);
std::wstring MyPath(L"/TiVoConnect?Command=QueryContainer\&Container="); MyPath.append(szHostName);
//std::wstring MyPath(L"/TiVoConnect?Command=QueryContainer\&Container=%2FNowPlaying");
std::wstring MyPlatform(std::wstring_convert<std::codecvt_utf8<wchar_t>>().from_bytes(myServer.m_platform));
std::wstring MyVersion(std::wstring_convert<std::codecvt_utf8<wchar_t>>().from_bytes(myServer.m_swversion));
std::wstring MyID(std::wstring_convert<std::codecvt_utf8<wchar_t>>().from_bytes(myServer.m_identity));
// These Keys are copied from what a real TiVo registers. These Values should be set based on the same Items I'm putting in the beacon
std::vector<PCWSTR> keys;
std::vector<PCWSTR> values;
keys.push_back(L"protocol"); values.push_back(L"http");
keys.push_back(L"path"); values.push_back(MyPath.c_str());
keys.push_back(L"swversion"); values.push_back(MyVersion.c_str());
keys.push_back(L"platform"); values.push_back(MyPlatform.c_str());
//keys.push_back(L"platform"); values.push_back(L"pc/WimTiVoServer");
keys.push_back(L"TSN"); values.push_back(MyID.c_str());
auto MyServiceInstancePtr = DnsServiceConstructInstance(
MyServiceName.c_str(),
MyHostName.c_str(),
nullptr,
nullptr,
ntohs(saServer->sin_port),
0,
0,
keys.size(),
&keys[0],
&values[0]
);
DNS_SERVICE_REGISTER_REQUEST rd = { 0 };
rd.Version = DNS_QUERY_REQUEST_VERSION1;
rd.InterfaceIndex = 0;
rd.pServiceInstance = MyServiceInstancePtr;
rd.pRegisterCompletionCallback = &DnsServiceRegisterComplete;
rd.pQueryContext = nullptr;
rd.hCredentials = 0;
rd.unicastEnabled = FALSE; // true if the DNS protocol should be used to advertise the service; false if the mDNS protocol should be used.
if (enable)
{
auto mDNSReturn = DnsServiceRegister(&rd, nullptr);
if (bConsoleExists)
std::cout << "[" << getTimeISO8601() << "] DnsServiceRegister(" << mDNSReturn << ") DNS_REQUEST_PENDING=" << DNS_REQUEST_PENDING << std::endl;
}
else
{
auto mDNSReturn = DnsServiceDeRegister(&rd, nullptr);
if (bConsoleExists)
std::cout << "[" << getTimeISO8601() << "] DnsServiceRegister(" << mDNSReturn << ") DNS_REQUEST_PENDING=" << DNS_REQUEST_PENDING << std::endl;
}
DnsServiceFreeInstance(MyServiceInstancePtr);
}
}
return;
}
When I create the Instance, I need to supply my hostname in the format HOSTNAME.local. I was initially trying to use the GetComputerNameEx function to get the ComputerNameDnsFullyQualified but my computer was configured as part of a domain and I was getting HOSTNAME.DOMAIN.local. I switched to get ComputerNameDnsHostname and manually add the .local.
I’m supplying nullptr for both the IPv6 and IPv4 addresses. The API seems to properly supply both the entries in the advertising. I’ve found that my machine reports its IPv6 address, while neither of my TiVos do.
WimsWorld 