My partner, Owen, and I are both into phones which serve a particular niche: I’ve always been a fan of Blackberry-style physical keyboards, and he prefers a small phone. Unfortunately for us, the vast majority of phone manufacturers seem to be dedicated to a single, monolithic design: thin slabs which are resistant to repair, water resistant, fancy cameras, thin bezels, etc. Basically every manufacturer from FairPhone to OnePlus to Google to Apple is roughly following this same vision for what makes a “high-end” phone, leaving niche users to be forgotten.

So, when I found UniHertz, it really felt like a breath of fresh air. Not only are they dedicated specifically to niche phones, offering both a line of phones that set records for their small size and a line of devices with BlackBerry- style physical keyboards, their phones also check other boxes: they disassemble with only a screwdriver and don’t have a strong focus on thinness.

So, a couple years ago I purchased a Unihertz Titan. It worked while it did. When the USB-C port went out and I was stuck with only wireless charging, I managed to convince them to sell me the part rather than have to ship it back to China for them to repair it. Unfortunately, that phone died in its sleep a few months later.

The Jelly 2

I never bothered to repair the Titan, but when Owen’s phone died, he asked me if I knew where he could find a small phone. I half-jokingly sent him a link to the page for the Unihertz Jelly 2, and he loved it. It honestly did well, and was fun to show off to people who had never seen an Android phone so small. However, it got dropped in a bucket of paint, making the speaker very hard to hear, and the screen had cracked some time ago. So, after a couple years of using the Jelly 2, he asked for an Atom L for his birthday.

The Atom L, and the Attack

The Atom L came in, and Owen swapped his SIM in and set it up. A little while later, I went to send him a message about something, and saw the “safety number verification” message in our signal chat.

Safety Number Mismatch message

I don’t always make a stink about verifying the safety numbers of someone who I know is in the next room over changing their SIMs, but I was feeling extra paranoid that day, and what happened next makes me feel like maybe we all should be a little more paranoid about our Signal verification codes. He pulled up his verification code, I pulled up mine, and, barely paying attention, I scanned the QR code from my phone…

Owen’s verification failed

Wait, what?

…that shouldn’t happen. None of us should have any idea what that screen looks like, unless we’re working directly on developing Signal’s Android application. The only thing any Signal user sees should be a nice green checkmark. I ran it the other way around, and it was the same. We puzzled on this for a bit… what could be happening? After a while, we decided to try the old phone.

The Jelly 2 shows verification failed

Two phones, failing to verify their Signal encryption keys? Could there be a problem on Signal’s end? I sent their customer service a message through their web site, both during the week after Christmas and again during the first week of February. As I’ve been suffering from long-covid this whole time, it took me a very long time to fully research and report this, and, not to spoil the story, I still don’t feel like I’ve found anything meaningful, but what can I do besides stop using Signal? That’s not really an option, my whole family is on Signal.

But what if it’s the manufacturer?

The next thing we did was to put Owen’s SIM into my phone1, and attempted to verify Owen’s account against another friend2. That succeeded, but we didn’t take screenshots of that.

Debug logs

Both logs are after clearing all app data, then signing in to the account and attempting verification.

Packet Capture

Setup

I went down a bunch of rabbit holes trying to get packet capture going3. I tried installing OPNSense, but didn’t have the right hardware. I tried setting up a laptop as an access point, but never got that working. I think part of that was the hardware as well, but it could’ve been that I didn’t know what I was doing also.

Finally, I settled on a VPN. I’ve been meaning to set up a Wireguard network anyway, so here we go. I set up a machine as a wireguard server. Click the ▶ to the left of this paragraph to see the Nix config I used.
# /etc/nixos/services/wg-server.nix
{pkgs, ...}: let 
externalInterface = "enp4s0f1";
serverIP = "10.100.0.1/24";
subnet = "10.100.0.0/24";
in {
  networking.nat.enable = true;
  networking.nat.externalInterface = externalInterface;
  networking.nat.internalInterfaces = [ "wg0" ];
  networking.firewall = {
    allowedUDPPorts = [ 51820 ];
  };

  networking.wireguard.interfaces = {
    # "wg0" is the network interface name. You can name the interface arbitrarily.
    wg0 = {
      # Determines the IP address and subnet of the server's end of the tunnel interface.
      ips = [ serverIP ];

      # The port that WireGuard listens to. Must be accessible by the client.
      listenPort = 51820;

      # This allows the wireguard server to route your traffic to the internet and hence be like a VPN
      # For this to work you have to set the dnsserver IP of your router (or dnsserver of choice) in your clients
      postSetup = ''
        ${pkgs.iptables}/bin/iptables -t nat -A POSTROUTING -s ${subnet} -o eth0 -j MASQUERADE
      '';

      # This undoes the above command
      postShutdown = ''
        ${pkgs.iptables}/bin/iptables -t nat -D POSTROUTING -s ${subnet} -o eth0 -j MASQUERADE
      '';

      # Path to the private key file.
      privateKeyFile = "/etc/srv/wg/privateKey";
      generatePrivateKeyFile = true;
    };
  };
}
Then, I created a config file for each phone, so that I could easily add them the the network via QR code. Click the ▶ to the left of this paragraph for an example.
# jelly.wg.conf
[Interface]
PrivateKey = {MY_PRIVATE_KEY}
Address = 10.100.0.4/24
DNS = 10.100.0.1
 
[Peer]
PublicKey = 2O9mV/asbJsrO2x7LAcVlmIOmV/0jATNOrBHh/bCrGc=
AllowedIPs = 0.0.0.0/0
Endpoint = 192.168.1.20:51820
Then, add the peers to the server’s Nix config…
# /etc/configuration/services/wg-peers.nix
{...}: {
  networking = {
    hosts = {
      "10.100.0.4" = ["atom"];
      "10.100.0.5" = ["jelly"];
    };
    wireguard.interfaces.wg0.peers = [
      {
        # Atom
        publicKey = "uHeZpGwXSRA4bS2VcDLnJQYxs+1iOLOixV2FHc3reVg=";
        allowedIPs = [ "10.100.0.4/32" ];
      }
      {
        # Jelly
        publicKey = "QgyV8W4OIobnG+sx1zjX+nEEUev72J+VtiIddxxX8Cc=";
        allowedIPs = [ "10.100.0.5/32" ];
      }
    ];
  };
}

…generate the QR codes…

$ qrencode -o atom.wg.conf.png atom.wg.conf
$ qrencode -o jelly.wg.conf.png jelly.wg.conf

…spread the config around, and we’re online, piping our traffic through a machine we can run tcpdump on! Finally!

Analyzing the Captured Packets

With the VPN set up, I ran the following command4

# tcpdump -i wg0 -w jelly-test.pcap
tcpdump: listening on wg0, link-type RAW (Raw IP), snapshot length 262144 bytes

And went through the Signal verification process again… and… verified? I don’t know why. You can find the packet capture file of that verification here. We tried it again, and this time, it failed. We’ll be looking at the packets from the second try, when it failed. I think this is a sign that we’re looking at some sort of glitch, at least, I hope. It is interesting to note that at least the first set of 5 digits in the verification code remained the same between the two tries, but at least the final 5-digit set differed.

Let’s start by taking a look at the DNS queries: which domain names were queried for, the IPs that map to those, and whether there are any IPs which were connected to which don’t map to a DNS record we have a query for. To do this I started off in WireShark, but pretty quickly realized I’d feel more at home in PyShark.

Python script used to generate below table entries
from pyshark import FileCapture
cap = list(FileCapture('jelly-test.pcap'))
for packet in cap:
    if 'dns' in packet and (a := packet.dns.get('a')):
        print('|', end=' ')
        print('A', packet.dns.resp_name, a, sep=' | ', end=' |\n')
for packet in cap:
    if 'dns' in packet and (cname := packet.dns.get('cname')):
        print('|', end=' ')
        print('CNAME', packet.dns.resp_name, cname, sep=' | ', end=' |\n')
Record Type Query (domain) Response (IP address(es))
A update.googleapis.com 142.251.40.227
A firebaseinstallations.googleapis.com 142.250.80.10
A chat.signal.org 13.248.212.111
A svr2.signal.org 20.119.62.85
A userlocation.googleapis.com 142.250.80.10
A storage.signal.org 142.250.65.211
A cdn.signal.org 18.238.49.106
A cdsi.signal.org 40.122.45.194
A svr2.signal.org 20.119.62.85
A sfu.voip.signal.org 34.36.20.21
A android.clients.google.com 142.250.65.206
A play.googleapis.com 142.250.65.202
A android.googleapis.com 142.250.72.106
CNAME svr2.signal.org svr2.trafficmanager.net
CNAME storage.signal.org ghs.googlehosted.com
CNAME cdn.signal.org d83eunklitikj.cloudfront.net
CNAME svr2.signal.org svr2.trafficmanager.net
CNAME android.clients.google.com android.l.google.com

Well, nothing suspicious there. Lets see if there were any IPs which were connected to which weren’t on that list of IPs that were queried for.

Python script used to gather IPs not queried for
known_ips = set(
    a
    for packet in cap
    if 'dns' in packet and (a := packet.dns.get('a'))
)
known_ips.add('10.100.0.5') # The local IP
known_ips.add('192.168.1.2') # A chromcast
known_ips.add('9.9.9.9') # configured DNS server
for packet in cap:
    if packet.ip.src not in known_ips:
        unknown_sources.add(packet.ip.src)
    if packet.ip.dst not in known_ips:
        unknown_destinations.add(packet.ip.dst)

This gives us two sets of IPs — the sources:

  • 18.238.49.6
  • 104.18.27.44
  • 104.18.26.44
  • 142.251.32.106
  • 76.223.92.165
  • 142.250.72.110
  • 18.238.49.66
  • 104.20.200.37
  • 142.250.31.188
  • 142.250.81.234

and the destinations:

  • 18.238.49.6
  • 104.18.27.44
  • 104.18.26.44
  • 142.251.32.106
  • 76.223.92.165
  • 142.250.72.110
  • 10.100.0.3
  • 18.238.49.66
  • 142.250.31.188
  • 142.250.81.234

Most of these are held in common between the two lists:

>>> unknown_destinations.difference(unknown_sources)
{'10.100.0.3'}
>>> unknown_sources.difference(unknown_destinations)
{'104.20.200.37'}

So, let’s look them up.

$ dig -x 104.20.200.37 | grep -A1 AUTHORITY | tail -n1
20.104.in-addr.arpa.    2271    IN      SOA     cruz.ns.cloudflare.com. dns.cloudflare.com. 2288625501 10000 2400 604800 3600

Okay, so we received a packet from cloudflare. Let’s take a closer look at that packet.

>>> for packet in caplist:
        if packet.ip.src == '104.20.200.37':
            print(packet.ip.dst)
10.100.0.3
10.100.0.3
10.100.0.3
10.100.0.3

Ah, so those are all junk, irrelevant to the test (they were between my phone and…who knows?). Lets look at the rest of them.

$ for ip in 18.238.49.6 104.18.27.44 104.18.26.44 142.251.32.106 76.223.92.165 142.250.72.110 18.238.49.66 142.250.31.188 142.250.81.234; do
$   printf "$ip: "; dig -x $ip | grep -A1 AUTHORITY | tail -n1
$ done
18.238.49.6: 
104.18.27.44: 18.104.in-addr.arpa.      3600    IN      SOA     cruz.ns.cloudflare.com. dns.cloudflare.com. 2288625505 10000 2400 604800 3600
104.18.26.44: 18.104.in-addr.arpa.      3600    IN      SOA     cruz.ns.cloudflare.com. dns.cloudflare.com. 2288625505 10000 2400 604800 3600
142.251.32.106: 
76.223.92.165: 
142.250.72.110: 
18.238.49.66: 
142.250.31.188:

Hmm, interesting. Lets see…

$ dig -x 18.238.49.6
; <<>> DiG 9.18.19 <<>> -x 18.238.49.6
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;6.49.238.18.in-addr.arpa.      IN      PTR

;; ANSWER SECTION:
6.49.238.18.in-addr.arpa. 82475 IN      PTR     server-18-238-49-6.jfk52.r.cloudfront.net.

;; Query time: 27 msec
;; SERVER: 10.64.0.1#53(10.64.0.1) (UDP)
;; WHEN: Sun Jan 21 15:56:19 EST 2024
;; MSG SIZE  rcvd: 108

Ah, I see, so we can fill that all in with

$ for ip in 18.238.49.6 104.18.27.44 104.18.26.44 142.251.32.106 76.223.92.165 142.250.72.110 18.238.49.66 142.250.31.188 142.250.81.234; do
$   printf "$ip: "; dig -x $ip | grep -A1 ANSWER | tail -n1   
$ done
18.238.49.6: 6.49.238.18.in-addr.arpa. 82593    IN      PTR     server-18-238-49-6.jfk52.r.cloudfront.net.
104.18.27.44: 
104.18.26.44: 
142.251.32.106: 106.32.251.142.in-addr.arpa. 86268 IN   PTR     lga25s77-in-f10.1e100.net.
76.223.92.165: 165.92.223.76.in-addr.arpa. 168  IN      PTR     ac88393aca5853df7.awsglobalaccelerator.com.
142.250.72.110: 110.72.250.142.in-addr.arpa. 86268 IN   PTR     lga34s32-in-f14.1e100.net.
18.238.49.66: 66.49.238.18.in-addr.arpa. 82594 IN       PTR     server-18-238-49-66.jfk52.r.cloudfront.net.
142.250.31.188: 188.31.250.142.in-addr.arpa. 3468 IN    PTR     bj-in-f188.1e100.net.
142.250.81.234: 234.81.250.142.in-addr.arpa. 86269 IN   PTR     lga25s74-in-f10.1e100.net.

That leaves us with this table of IP addresses and hostnames, none of which look suspicious:

IP Hostname Org
142.251.40.227 update.googleapis.com Google
142.250.80.10 firebaseinstallations.googleapis.com Google
13.248.212.111 chat.signal.org Signal
20.119.62.85 svr2.signal.org Signal
142.250.80.10 userlocation.googleapis.com Google
142.250.65.211 storage.signal.org Signal
18.238.49.106 cdn.signal.org Signal
40.122.45.194 cdsi.signal.org Signal
20.119.62.85 svr2.signal.org Signal
34.36.20.21 sfu.voip.signal.org Signal
142.250.65.206 android.clients.google.com Google
142.250.65.202 play.googleapis.com Google
142.250.72.106 android.googleapis.com Google
18.238.49.6 dns.cloudflare.com Cloudflare DNS
104.18.27.44 dns.cloudflare.com Cloudflare DNS
104.18.26.44 dns.cloudflare.com Cloudflare DNS
142.251.32.106 lga25s77-in-f10.1e100.net Google
76.223.92.165 ac88393aca5853df7.awsglobalaccelerator.com AWS
142.250.72.110 lga34s32-in-f14.1e100.net Google
18.238.49.66 server-18-238-49-66.jfk52.r.cloudfront.net Cloudfront
142.250.31.188 bj-in-f188.1e100.net Google
142.250.81.234 lga25s74-in-f10.1e100.net Google

For posterity’s sake, I ran the same test on the newer Atom L. Here are those logs if you’d like to download them and analyze them yourself. At first glance, I don’t see anything.

The same Python script as above, used to generate below table entries
from pyshark import FileCapture
cap = list(FileCapture('atom-test.pcap'))
for packet in cap:
    if 'dns' in packet and (a := packet.dns.get('a')):
        print('|', end=' ')
        print('A', packet.dns.resp_name, a, sep=' | ', end=' |\n')
for packet in cap:
    if 'dns' in packet and (cname := packet.dns.get('cname')):
        print('|', end=' ')
        print('CNAME', packet.dns.resp_name, cname, sep=' | ', end=' |\n')
Record Type Url IP Address
A chat.signal.org 13.248.212.111
A updates2.signal.org 104.18.27.44
A svr2.signal.org 20.119.62.85
A storage.signal.org 142.251.32.115
A cdn.signal.org 18.238.49.6
A cdsi.signal.org 40.122.45.194
A svr2.signal.org 20.119.62.85
A cdn.signal.org 18.238.49.90
A cdn.signal.org 18.238.49.90
CNAME updates2.signal.org updates2.signal.org.cdn.cloudflare.net
CNAME svr2.signal.org svr2.trafficmanager.net
CNAME storage.signal.org ghs.googlehosted.com
CNAME cdn.signal.org d83eunklitikj.cloudfront.net
CNAME svr2.signal.org svr2.trafficmanager.net
CNAME cdn.signal.org d83eunklitikj.cloudfront.net
CNAME cdn.signal.org d83eunklitikj.cloudfront.net

And to be sure, lets reverse-DNS those IPs:

$ for ip in 13.248.212.111 104.18.27.44 20.119.62.85 142.251.32.115 18.238.49.6 40.122.45.194 20.119.62.85 18.238.49.90 18.238.49.90; do
    printf "$ip: "
    dig -x $ip | egrep -A1 'ANSWER|AUTHORITY' | tail -n1
  done
13.248.212.111: 111.212.248.13.in-addr.arpa. 239 IN     PTR     ac88393aca5853df7.awsglobalaccelerator.com.
104.18.27.44: 18.104.in-addr.arpa.      3539    IN      SOA     cruz.ns.cloudflare.com. dns.cloudflare.com. 2288625505 10000 2400 604800 3600
20.119.62.85: 62.119.20.in-addr.arpa.   240     IN      SOA     ns1-07.azure-dns.com. azuredns-hostmaster.microsoft.com. 1 3600 300 2419200 300
142.251.32.115: 115.32.251.142.in-addr.arpa. 86340 IN   PTR     lga25s77-in-f19.1e100.net.
18.238.49.6: 6.49.238.18.in-addr.arpa. 82666    IN      PTR     server-18-238-49-6.jfk52.r.cloudfront.net.
40.122.45.194: 45.122.40.in-addr.arpa.  3541    IN      SOA     ns1-201.azure-dns.com. msnhst.microsoft.com. 1 900 300 604800 3600
20.119.62.85: 62.119.20.in-addr.arpa.   240     IN      SOA     ns1-07.azure-dns.com. azuredns-hostmaster.microsoft.com. 1 3600 300 2419200 300
18.238.49.90: 90.49.238.18.in-addr.arpa. 82713 IN       PTR     server-18-238-49-90.jfk52.r.cloudfront.net.
18.238.49.90: 90.49.238.18.in-addr.arpa. 82713 IN       PTR     server-18-238-49-90.jfk52.r.cloudfront.net.

And, again, all we have is traffic to CDNs, which doesn’t tell me anything. Like I said, this all looks pretty innocuous. Maybe someone who knows more will be able to determine more information. Even if there’s not a nefarious thing going on here, until this is resolved, I can’t rely on Signal as a secure way to communicate with my partner, and I don’t really know what else to do.

Conclusion

To summarize, we have two phones from one manufacturer which Signal verification fails on. This raises suspicions of a Man-in-the-Middle attack, but I was unable to gather evidence to support that hypothesis. I’m still left with pressing questions:

  • …what could be causing this?
  • …are there other small phones my partner could use which wouldn’t be suspected to be spying on our private conversations?

I can be reached via PGP encrypted email at scott@techwork.zone5, via Matrix at @dscottboggs:matrix.org, or by Signal6.


Edit: 2024-03-06

I received a Signal message on February 29 from someone who said they “also had these issues with a friend but lacked the time or will to investigate it further”. They also said it happened about two years ago, and when I asked what hardware this had been happening between, they said it was an older Samsung phone.

Unfortunately, that doesn’t tell us a lot diagnostically, besides “this has happened before”. While I’m updating the article, I thought I’d save the readers the time of looking up the hardware specifications for the phone: both the Jelly 2 and Atom L are based around a MediaTek Helio P60 SoC.

Lastly, if you’ve made it this far, please share this around. I haven’t received a lot of attention on this issue, and still haven’t heard anything from Signal. I just made my third attempt at contacting them, maybe I’ll hear back this time.

A Screenshot of the Signal help form


  1. My phone is a Google Pixel 6 running GrapheneOS ↩︎

  2. Our friend has a stock OnePlus 9 Pro ↩︎

  3. I had always somewhat intentionally avoided looking at packet captures before, as I’ve heard it can be anxiety inducing. It did turn out to be, at first, but after a bit of digging, it turned out to be a bit reassuring, at least when it comes to devices that I control personally. ↩︎

  4. Shout out to Julia Evans for her awesome zine on the essentials of tcpdump↩︎

  5. PGP fingerprint 9478 FED0 F32D 80B2 AD38 CD70 E549 48CA 806D 4D26 ↩︎

  6. This article didn’t originally have my signal chat link, I added it after updating to the new Beta version the morning after posting the original version. ↩︎