HFGCS · Structural heatmaps

How to Read These Heatmaps

There is one heatmap page for each message length from 17 to 292 characters. Each row is one message. A character is coloured when it is part of a pattern that passed its test, and pale when the pattern came close.

Last updated 261005.

What are EAMs?

EAM stands for Emergency Action Message. EAMs are coded messages broadcast over high-frequency radio by the US military's High Frequency Global Communications System (HFGCS). Each one is a string of capital letters and digits. They are sent in the clear, but what they mean is encoded. Listeners log them, and the heatmaps are built from that log (SHORTWAVES).

The rows

Each row is one message text. A message that was rebroadcast appears once, and the date, time and callsign above it are from its first row in the log. Rows are sorted by group, oldest first. The Chronological view puts every row in time order.

Each message is put in a group by its prefix, the first two characters. The PR_GROUPS table in the log lists which group used each prefix from which year, and the latest entry at or before the year of the broadcast decides. A prefix with no entry goes to Unknown.

Group 1 Group 2 Group 3 Group 4 Group 5 Unknown

Every test runs separately inside each group at each length. Groups never share evidence, and nothing is borrowed from other lengths.

What is left out

As of 261005, 15,211 rows pass these filters.

Colours

Each colour is one kind of pattern. How bright it is shows how common the pattern is in the group (see Brightness).

221116 06:22 CHOPHOUSE · Group 4 · 142 characters, positions 1–45 as coloured on its page
PDZQVPLQA3VSJOX1
TJ7BZSIHSAMHJEI212430
AIJ6DHLAKA47777324245

Positions 1 and 2 (PD) are the prefix and stay uncoloured. Positions 21 and 24 carry the same character (S), in amber. Positions 30 and 32 match too (I), in blue, and 42–45 is a run (7777), in red.

The significance test

With hundreds of thousands of position pairs, some will match often by chance alone. So each kind of pattern is corrected for every test of its kind across the whole dataset (a Bonferroni correction): the chance probability a pattern must beat is 0.05 divided by the number of tests. As of 261005 the counts are:

The chance that even one pair or run that passed, on any page, is luck is at most 5%. The same holds for each of the other kinds.

The cost is that small groups need near-total agreement. If two positions match by chance 3% of the time, a match in all 4 of 4 messages has a chance of 0.03⁴, about 1 in 1.2 million: not enough, so it shows pale. All 5 of 5 is 0.03⁵, about 1 in 41 million: solid. Groups are never pooled across lengths, so a pattern known from other lengths still shows pale where its group is small.

Not every colour uses this test:

Pale colours

KKKPale

A pale pair or run missed the dataset-wide bar, but would be expected by chance fewer than 2 times among all the tests of its kind on its page. Some pale chips are luck. Sub-formats have a pale tier too; mirror blocks do not.

Every pale colour is set to the same low brightness against the dark page, about 0.1, with light text. Pale chips only go on characters that have no other pattern.

Pairs

IAIBlueSIHSAmber

For each pair of positions in a group, the test counts the messages where both positions carry the same character; at least 3 are needed. Chance is the probability that two characters of that group match at random: each character's share of the group's text, squared and summed, about 3% for Group 4. A pair that passes is coloured blue.

A pair with exactly two characters between its ends, X??X, is coloured amber instead. The test is the same; only the colour differs.

A mirror block is a block of 2 to 5 characters that each message repeats at the same two places, whatever the characters are. It needs at least 3 messages (2 if the group has only 2) and is tested as one block. Its pairs are coloured blue, and clicking one lights the matching block in the other messages.

Runs

7777Red33Coral

A run of 4 or more identical characters is coloured red in every message. It is at full brightness inside a run zone and at least 0.75 outside one.

A run zone is a stretch where the group's runs line up. Either of two things makes one: a run of 2 to 5 at that start that passes the dataset-wide test (in at least 2 messages), or at least 15% of the group's messages (and at least 2) having a run of 4 or more there.

A run of 2 or 3 is coloured coral only where at least 2 of its characters fall in a run zone. A run that recurs but misses the bar shows pale coral, only on uncoloured characters in messages that have it.

Repeats inside a message

K4JTeal

A message can repeat part of itself. Three kinds are coloured teal:

An exact or gapped repeat is coloured only if it is too long to be chance in a message of that length. The expected number of chance repeats, L² ÷ (2 × 26ᵏ), must be below 0.05, where L is the message length and k the number of matching characters. That means 3 or more characters at 30, 4 or more at 142 and 5 or more at 292. A shorter one stays uncoloured, unless the same characters are part of a pair or block that passed.

Alternative forms

33CoralSIHSAmberIAIBlue

Some structures have a minority form. Long Group 4 messages without the same character at 21 and 24 often have a doubled character at 23–24 instead (??XX). On the 142 page, 5 of the 6 messages without it have the double.

The test takes a structure that passed (a pair 3 to 6 apart, or a run zone of 4 or more), keeps only the messages that lack it, and re-tests the pairs within 3 positions of it on those messages alone. The group needs at least 5 messages. A form that passes is coloured by its shape: coral for a double, amber for X??X, blue otherwise. Its brightness follows how many of those messages carry it.

Sub-formats

NRNViolet

A sub-format is a set of pairs that all hold in the same subset of the group's messages: at least 3 messages, and fewer than three quarters of the group. It needs at least 4 pairs, at least 2 of them independent (a stretch of neighbouring columns counts once). It is looked for in groups of 3 to 80 messages and coloured violet in its member messages only, on characters no other pattern holds. Click one to see the members.

Format changes (2A and 2B)

Some groups changed format partway through. For each group with at least 8 messages, the test orders the messages by first airing and finds the split where the matching positions change most. The split counts if a shuffle test puts it below 0.05 ÷ 52, if each side with fewer than 5 messages is at least 75% consistent with at most 1 exception, and if the two sides share less than a quarter of their structural columns. If they share more, it is a variant and the group stays whole.

A group with no split gets one more test at a fixed date, 2024-10-01 00:00 UTC, when the 30 character Group 1 format changed. It passes if the shuffle test holds at that date, or if 3 or more positions have a doubled character in every message on one side and in none on the other.

A split group is shown as two parts, such as 2A (before) and 2B (after), each tested on its own. Both keep the group colour. The splits are at 38 (Group 3), and at 39, 43 and 55 (Group 2).

Accent shades

HHHMauve, rose, periwinkle

Between two run zones, several pairs of one colour can overlap and blur together. When they do, the second and third get an accent shade: mauve, rose or periwinkle. A pair keeps its shade in every row. The pattern and its test are unchanged.

Brightness

Brightness shows how common a position's pattern is in its group. For each position, the page counts how many of the group's messages have a pattern there. The fill is 0.4 + 0.6 × (log(c + 1) ÷ log(m + 1))³, where c is that count and m the highest count in the group, and never below 0.15. Only the fill fades; the character stays readable.

So a dim solid chip is a pattern that passed, in a column that few of the group's messages share. It does not mean a weaker result. Runs are the exception: full brightness inside a run zone, at least 0.75 outside.

Which colour wins

When patterns overlap on one character, it shows one colour, in this order: run of 4 or more, short run, gapped repeat or motif, exact repeat, pair. There are two exceptions. An exact repeat that is also the end of a pair shows as the pair, unless it sits inside a mirrored stretch of 2 or more pairs that passed. A gapped repeat inside such a stretch shows as an exact repeat. Pale colours and sub-formats only go on characters with no other pattern.

What is set by hand

No pattern is placed by hand. The X??X at 21–24 on long Group 4 messages comes out of the pair test like any other pair: positions 21 and 24 carry the same character far more often than chance. What is set by hand:

The bar

The bar above the rows has one cell per position and stays at the top as you scroll. Its orange is the average fill of the coloured characters in that column over the rows on screen: their brightness added up and divided by the number of rows, up to 1. It is recalculated when groups are filtered. Hover over a cell for its value.

Using the pages

The older version

The older version has no dataset-wide test. A pair is hot if the same character is at both positions in at least 30% of the group's messages (and at least 3). Repeats are faded by a chance score instead of removed. Positions 1 and 2 are included. Its colours are pink for runs of 4 or more, rose for runs of 2 or 3, green for gapped repeats, blue for exact repeats and purple for pairs.