Shortwave Trading | Part I | The West Chicago Tower Mystery

Since 2014 this blog has extensively covered the wireless networks built by high-frequency trading (HFT) firms or network providers to reduce latencies between the different exchanges around the world (market makers need fast connectivity to manage risk, news traders also need to be fast, etc.). This epic investigation on microwave, which started with HFT in my backyard, will be fully reported in a book I’m currently writing (in French for now). As I’m quite busy with this writing (and other/more interesting matters about market structure), I didn’t really have the time to check out what I have been hearing about “shortwave” or “high frequency” radio. This is the way high-frequency trading firms may use shortwave radio to directly connect widely-separated locations (in short, traders are willing to use shortwave to cross oceans with less latency than any fiber – like Hibernia).
But recently I got more intel about the situation (and some fun anecdotes). With some help from the US, I found that a firm purchased a field for more than 1$M to build towers and antennas; with some help from the EU, I got hints about Germany; and I dug into UK public records.
I even met, last March in Amsterdam, people involved in those projects. Not surprisingly, at least five HFT/market making firms showed up behind the shell companies/names they use to hide. The usual suspects. Above all, I have been contacted recently by someone from Chicago, Bob, who decided to investigate the “shortwave” networks in his backyard. Today I’m pleased to host Bob as a new guest writer on this blog. This first part of the “Shortwave Trading” series is released at the same time Bob is talking about what he found at the STAC Summit in Chicago. Next parts will follow soon.

I’m Bob Van Valzah and I’m on a “gardening leave” between jobs working as a Performance Engineer for high-frequency trading firms in Chicago. I recently stumbled onto the first evidence of shortwave trading at a site in West Chicago, Illinois and then used Federal Communications Commission (FCC) database research techniques pioneered by the owner of this blog to find two more sites. My non-disclosure agreements with employers would typically prevent me from talking about this, but my trading-related discoveries are my own while on gardening leave. The results of my research may be of interest here and there, so I offered to do some guest posting on this blog. I’ll have to go quiet again in July when I head back to work.

It might be reasonable to assume that “West Chicago” was just a western part of Chicago, Illinois. In fact, it’s a city in its own right, twenty some miles west of the big city. A typical weekend will find me cycling down a bike path that runs nearby. In March, muddy path conditions caused me to detour through an industrial park in West Chicago where the West Chicago Tower Mystery began. I looked up from the handlebars and saw this tower.

WestChicagoTower

Mysterious West Chicago Tower, March 10, 2018.

I’m an amateur radio operator, or “ham,” (KE9YQ) and those four big antennas look a lot like ones that a ham would use to talk across an ocean. I’ll just call them shortwave antennas here, but practitioners of aluminum feng shui (antenna design) will recognize them as log periodic. There’s also a microwave dish antenna between the top two shortwave antennas. This much I could tell just by looking.

Most of the radio towers you see each day are cell towers. There are 215,000 of them in the US. They differ from other radio towers in that they will almost always have one or more triangular structures with three or more directional antennas on each side of the triangle. A grid of cell towers covers an area with roughly hexagonal cells so that you get a good signal everywhere.

The West Chicago tower had no triangular structure, so I was pretty sure it wasn’t a cell tower. And these antennas were huge compared to regular cell tower antennas. I could literally see them 1/2 mile away. Some cell towers do have microwave dish antennas when it’s hard to bring fiber to a site, but shortwave antennas didn’t seem to go together with microwave in my mind.

Mystery #1: If this wasn’t a cell tower, what was it?

With my curiosity up, I pedaled to the base of the tower and took a look around. Like all cell towers, there’s a barbed wire perimeter fence and I could see a power meter where electricity enters the site. The panel allowed for up to four power meters because it’s common for two or more carriers to share a single site. There’s also a master power switch for each carrier so that they can shut off their juice for maintenance without knocking the other carriers off the air.

WestChicagoTowerBase

The tower base and perimeter fence.

WestChicagoPowerMeter

The only power meter for the site.

I walked right up to the fence and zoomed in on the power meter. It said “U.S. Cellular.” They are indeed a regional cellular carrier, but they only serve 23 U.S. states and wouldn’t want their signals crossing any oceans.

Mystery #2: Why would a regional cellular carrier pay for power to cross oceans?
Riding on a hot day, I may stop for a drink in the shade of a cell tower, so I’ve seen what a lot of cell sites look like. There’s often a small building inside the perimeter fence to hold the equipment that runs the cell site. The ground inside the perimeter fence is typically coarsely crushed rock to prevent weeds from growing.

An odd thing about this “cell site” was that it had a garbage pile inside the fence. Cell sites are unmanned and don’t need much maintenance. Crews who do work there typically leave the site clean and take their garbage with them. There is no trash pickup at sites like this, so it seemed odd that there’d be cans full of lunch wrappers and cardboard boxes. If this was a cell site, it was maintained by much sloppier crews than the ones I’d typically seen.

One box in the garbage pile caught my eye so I zoomed in through the fence and got the best shot of the label I could with my iPhone. I had never heard of the company Ettus Research or the USRP X300 device. With all the cell sites in the world, you’d think they’d be made from off-the-shelf parts, not stuff cooked up in a research and development lab.

EttusResearchLabel

The label on an interesting box in the garbage pile.

Mystery #3: Why would a cell site need equipment from a research company?
Most any cell site with a tower will have a sign with a seven-digit number known as the FCC Antenna Structure Registration (ASR) number. If you were a cellular carrier out shopping for a good place to put your antennas, you could use this number to find the height of the tower and get other useful nuggets from the FCC database. I took a picture of the number since I thought it might come in handy when I got home.

ChangeOfOwnership

U.S. Cellular sold the tower in 2014.

Back home, I pulled up the FCC ASR database and found that the tower was indeed built by U.S. Cellular in 2009. Digging deeper, I also found a record that they had sold it in late 2014. This sort of transaction is pretty common in the business. As subscribers come and go, cellular carriers have to move their cell sites and keep adjusting their antenna heights to maintain good coverage everywhere.

Google Street View is a wonderful tool for looking at the way the world used to be. I found a view of the tower from August 2012 when it had already been picked clean of all cellular antennas (they have resale value!), with only the tell-tale triangular mounting structure remaining at the top of the tower.

West Chicago Tower 8:2012

The tower as it looked in August 2012.

So part of the history was becoming clear. The tower was built by U.S. Cellular in 2009, but fell into disuse sometime before 2012. It was eventually sold in late 2014.

As is common with cell sites these days, the tower site was sold to Vertical Bridge, an antenna site management company. Instead of owning antenna sites themselves, it now seems common for carriers to just lease space on towers at antenna sites owned by management companies. So that’s no real clue about current ownership or usage of the tower.

Mystery #4: Where is the other end of that microwave link?
Microwave links require a license for operation, so I returned to the FCC databases and looked for licenses associated with that tower. Each license gives the GPS coordinates of both ends of the link and the FCC has a helpful mapping function. It showed a direct link between the West Chicago tower and another tower right by the Chicago Mercantile Exchange! It was still just circumstantial evidence, but this was the first good clue that the West Chicago tower had something to do with trading. It was looking less and less like spooks were using the tower.

FCCPathMapAnnotated

FCC map of microwave connection between CME and West Chicago tower.

EttusResearchUSRP X300

Top-of-the-line software defined radio.

Mystery #5: What was in that box in the garbage pile?
Google helpfully explains that Ettus Research is the leading provider of Software Defined Radios (SDR). These are the devices you need to make any kind of custom radio you want. Just change the software and you can send voice, pictures, video, or trading information. The particular model in the box is their top-of-the line, with an FPGA, two 10 gigabit Ethernet ports, and a PCIe bus interface. The pair that was in that box cost nearly $10,000. You don’t need gear like this at a cell site.

Mystery #6: What could be interesting across an ocean?
I’m much better with the technology of trading rather than understanding how to make money with trading. But it seemed like a reasonable guess that prices on the markets at CME would be correlated with activity in European markets – let’s say, in Slough (UK) and Frankfurt (Germany). So I fired up Google Earth and plotted the lines that a radio signal would have to travel between West Chicago and the markets in London and Frankfurt.

WestChicagoLondonFrankfort

Shortest paths between West Chicago and European markets.

We are so used to seeing flattened projections of earth that it’s easy for a Chicagoan to think of Europe as being east of Chicago. But in fact the shortest way to Europe is northeast, at about a 45º angle.

WestChicagoEuropeAngle

For radio, Europe is northeast of Chicago.

This angle is important because those shortwave antennas on the tower are directional. If you know you want to talk to Europe, why send any of your signal toward Guam? Directional shortwave antennas focus your signal in the direction of the pointy end of the antenna. In the photo of the tower above, I’m standing right under the path where the top antenna is pointing.

At shortwave frequencies, it’s difficult to make antennas with a narrow beam width. London and Frankfurt are only about 5º apart when you’re aiming from Chicago, so one antenna should hit both cities. This raises the question of why the tower needed four shortwave antennas, all pointed in pretty much the same direction.

Some of the antennas on the tower focused their signal well, producing a more powerful signal, but only operate over a narrow range of shortwave frequencies. Others operate over a wide range of frequencies, but can’t produce a strong signal.

MarchToAprilChanges

The lower two antennas have been removed from the tower.

It also appeared that the second shortwave antenna up from the bottom had been damaged in a wind storm or perhaps poorly installed because four of its five elements had been knocked significantly out of the horizontal plane. Nonetheless, I was surprised when I visited site a few weeks later and found that the lower two antennas had been removed. So this site is being actively developed in the spring of 2018. They’re moving antennas around and they haven’t gotten rid of the garbage pile, so there’s still work to do.

Bringing it all together, I’ve discovered that this West Chicago tower has all the ingredients needed to make a low-latency transatlantic market connection. I’ve also found two other sites around Chicago with massive shortwave antennas pointed at Europe and microwave links to CME, so there’s more than one horse in this race. I’ll document these sites in a coming post. Licenses also exist on the east coast. I’ll be looking for them on trip in mid-June. FCC searches have also turned up licenses in Anchorage, Alaska which is on the path from CME to Tokyo [Sniper’s note: that reminds me I didn’t check recently what is going on around Go West]. Spotters are already at work looking for antennas around London and Frankfurt.

Somewhere around Slough (UK), March 2018

Shortwave is no panacea. It’s unreliable, expensive, and very low bandwidth. Think dialup speeds. But you can’t beat it for latency. I plan to go into more detail on shortwave tradeoffs in a future post. You might be wondering if anybody can receive their signal, perhaps with the possibility of getting an order to market faster. Yes, anybody can receive it, but the sender would be foolish not to encrypt it in a way that can’t be cracked. This is easily done without impacting performance.

A muddy bike path diverted me from my usual route and I happened upon the tower. Being a ham, I recognized the shortwave antennas. I could see the potential, having worked in trading. I’m on gardening leave so I can talk publicly about what I found related to trading – I’ve just assembled the public records, photos, maps, background info, and sprinkled in my analysis, with some inspiration from the way this blog investigated the microwave networks. The next post from me will describe the differences between microwave, shortwave, fiber, and satellite. Stay tuned.

Somewhere in Kane County (US), April 8th, 2018

Advertisements

“I will be back”

FOW Conference – Amsterdam – March 7th

Hello everybody. As pointed out by a couple of folks (hello Pedro) who attended my (too long) talk at Fow Trading Amsterdam two weeks ago, this blog has been outrageously silent far too long. Not that I don’t have interesting stories to write about (I have too much to tell), but this academic year has gone by very fast. It’s done now, classes are over and I have a few months to do research, to post here all the drafts I prepared, to write a new book about HFT, “The towers war” (title non-contractual) – I didn’t even write a single line… –, and to finish a proposal for a wider study/book I would like to do in the next three/four years. This proposal will be finished next Monday, so that next week I’ll be able to switch on again this blog (starting with an updated and developed version of my FOW talk – slides will be available).

In the meantime, if you want a reading, here is an English translation of a text I wrote two years ago as a foreword for a new edition of a very famous old dictionary, Dictionnaire historique et critique de Pierre Bayle. It’s all about data, data centers and coding, typography and regexps, and so on. That’s when I worked on this Arca studiorum data center (picture below) years ago that I understood the exchanges were now in data centers… so begun my investigation into the so-called “high-frequency trading”. This text is here. I hope you’ll enjoy it. See you next week to talk about canaries flying fast in Chicago.

Network Effects | Part II

The nondisclosure agreements have lapsed. The Chicago to New Jersey microwave arms race has converged to a few winners. Many of the early participants must now be eying the runaway success (and the glaring shortcomings) of a certain HFT-associated bestseller, and thinking,  yeah, “I could do that.”

It’s a fantastic story, after all, and it hasn’t really been told yet. It seems like there are two basic approaches. You could write a cinema-ready page-turner heavy on the skulduggery. Antennas knocked out of alignment the night before the jobs number. Unlicensed broadcasting on cognitive radio. Itinerant con men peddling futures on networks that will never exist. Or you could try to write a book with longer-term importance that draws the details within the larger context and paradoxes of the modern-day United States.

*

A colleague in the tech industry was recently recalling his first encounters with the real Flash Boys.

“It was some time in the summer of 2010 when the calls started coming in,” he said, “It came on suddenly, and then it ramped up. Fast. All through the fall and then into 2011. They’d come on site, and it was clear they weren’t your traditional telecom guys. They were a lot younger, for one. Sandals, cargo shorts, T-shirts. They flat-out refused to say what it was they were actually doing.

“They talked gain margins, modulation, propagation physics, et cetera, but in an oddly theoretical way. It was as if they’d just stayed up late mastering a textbook. Every single one of them wanted to know about regeneration latency. Up to then I’d never given any consideration to internal latencies. Normally, on the digital signal processing side, you’re interested in error correction, and doing it absolutely as well as you can. Extra microseconds had never mattered, period.

“They weren’t staying at the Super Eight. They’d limo out, keep the limo waiting all day. Once, I said to one of them, ‘Hey, let me give you a lift back to the airport.’ Turned out he wasn’t flying commercial. I dropped him off at general aviation. I saw that a G4 was waiting on the tarmac.”

*

In early 2011, I attended an event, known as a research “Jamboree” where everyone – that is, the large collection of physicists working in the building – stands up in rapid succession and gives a one-minute talk while showing pre-loaded PowerPoint slides. Having already given several of these talks, I knew that my previous modus operandi, which had consisted of earnestly trying to explain too much research, and then getting unceremoniously cut off, Gong Show style, was completely ineffective.

When my minute came up, I went up to the podium. My slide was a screenshot of a maze of numbers; an asof joined kdb+ table of equity quotes – sample data that I had been using while learning the terse Q language.

The one-minute timer started. I looked at the audience, keeping my expression entirely blank. Five, six, seven seconds ticked by awkwardly.

“One minute isn’t very much time,” I said, finally, breaking the silence, “but it’s an eternity in the highly secretive, highly profitable world of high-frequency trading.”

I pointed to the numbers on the slide. “This is a time series of offers to buy and sell shares of stock. What’s completely amazing is that the data structure holding these ticks has a nanosecond field. Every one of the 23,400 seconds in the trading day is now potentially divisible into a billion individual increments. Kind of staggering when you compare with the fact that Earth is 4.5 billion years old.”

“A decade ago, talking about nanosecond stamps on the trade blotter would have been absurd. In the Dot-Com boom, you had day traders flipping shares of Xilinx, and sometimes they even managed consistent profitability. The fastest human reaction times are of order one hundred and fifty million nanoseconds.”

“Speed of light is a foot per nanosecond. Einstein’s theory of special relativity is now an economic issue. A few more orders of magnitude decrease in latency, and general relativistic time dilation starts to matter. In the future, to get a competitive interest rate, you’ll need to live deep in a gravitational potential well, and you’ll need to have your bank out in flat space time.”

My minute was up. Everyone was staring at me like I’d arrived from outer space.

“Well, Uh, OK…” the moderator said, “That was something different. Our next speaker is going to tell us about the accretion of gas onto galaxies at redshift Z=2.”

*

 The next day, I ran into a colleague, I’ll call him Tim, in the hallway, “Were you serious?” he asked.

“More or less,” I said, “General Relativity – gravitational time dilation – that’s for sure playing no role in the market right now, but special relativity is relevant in the sense that the distinction between timelike and spacelike events definitely matters if you are back-testing a strategy against historical data.”

I told him about the scheme to use a microwave relay to beat Spread’s Network. “Apparently, they spent of order $300M to trench the fiber. Wireless could be more than two orders of magnitude less expensive…”

“I’m in,” he said.

*

Even at 1% the expense of Spread’s fiber, a clandestine wireless relay spanning a third of the continent presents a daunting project. The urgent question at the start of 2011 was whether anyone had gotten there first.

“If it’s being done already, maybe it’s sitting in plain sight. Say they’re using licensed spectrum. The Federal Communications Commission [FCC] might have a searchable database?”

They do indeed. 

The geodesic arc from Chicago to New York never strays far from 41 N, and the FCC has made it easy to query for entities that fit specified criteria within a relevant geographic range. A first pass through the filters produced a jumble of familiar and obscure names:

AB Services LLC

airBand Communications

AT&T Corp

Cellco Partnership

Clearwise Spectrum Holdings II & III, LLC

Comprehensive Wireless, LLC

ECHOSTAR CORPORATION

FELHC, Inc.

FiberTower Network Services Corp.

FiberTower Spectrum Holdings LLC

Fundamental Broadcasting LLC 

HISPANIC INFORMATION AND TELECOMMUNICATIONS NETWORK; INC.

IDT Spectrum; LLC

MCI Communications Services; Inc

METROPOLITAN AREA NETWORKS; INC.

MPX Inc.

New Cingular Wireless PCS; LLC

Norfolk Southern Railway Company

Northrop Grumman Systems Corp.

Open Range Communications

Telecom Transport Management; Inc.

TEXAS EASTERN COMMUNICATIONS INC

Thought Transmissions LLC

Towerstream Corp.

Trex Enterprises Corporation

Westwood One Radio Networks; Inc.

With the transect list in hand, it was straightforward to step through the firms and plot their antenna locations. A Chicago to New York line-of-sight-relay would presumably be obvious, even at a glance. First on the list was “AB Services”, but the FCC website had suddenly slowed to an infuriating crawl. We waited for nearly a minute. Finally a map appeared on the screen, triggering a mixture of awe and disappointment.

A quick back of the envelope calculation indicated that even with off-the-shelf radio latencies, AB Networks, as licensed in the FCC  database, was easily capable of beating Spread’s fiber. Google linked the LLC to Anton Kapela and Alex Pilosov, two gentlemen who, if nothing else, appeared to have a variety of marketable tech skills. A Wired article from 2008 reported on a presentation they’d given at the DefCon hacker conference: 

“…BGP eavesdropping has long been a theoretical weakness, but no one is known to have publicly demonstrated it until Anton ‘Tony’ Kapela, data center and network director at 5Nines Data, and Alex Pilosov, CEO of Pilosoft, showed their technique at the recent DefCon hacker conference. The pair successfully intercepted traffic bound for the conference network and redirected it to a system they controlled in New York before routing it back to DefCon in Las Vegas…”

Stepping through the list of licensees indicated that “Comprehensive Wireless”, “Fundamental Broadcasting” and, the creepily named “Thought Transmissions” had, starting a few months after AB’s appearance in September 2010, also licensed a network, using the different LLCs to help mask their intentions. Comprehensive-Fundamental-Thought had set up a more direct route, and if operative, would be even faster than AB’s. 

*

It was clear that an arms race was developing, with an end state winner destined to be a very straight network with very fast radios. To see how far along things were, we needed to look at tick data to discern how fast the information was propagating. Was anyone already up and already front-running Spread?

The Forbes article suggested that the fiber build was spurred by a trade involving “tiny discrepancies between futures contracts in Chicago and their underlying equities in New York.” Looking through the academic literature, a paper by Joel Hasbrouck, titled “Intraday Price Formation in U.S. Equity Index Markets” jumped out by virtue of having been cited hundreds of times. The abstract reads:

“The market for U.S. equity indexes presently comprises floor-traded index futures contracts, exchange-traded funds (ETFs), electronically traded, small-denomination futures contracts (E-minis), and sector ETFs that decompose the S&P 500 index into component industry portfolios. This paper empirically investigates price discovery in this environment. For the S&P 500 and Nasdaq-100 indexes, most of the price discovery occurs in the E-mini market. For the S&P 400 MidCap index, price discovery is shared between the regular futures contract and the ETF. The S&P 500 ETF contributes markedly to price discovery in the sector ETFs, but there are only minor effects in the reverse direction.”

So it appeared that we were in familiar territory! We were effectively faced with a particle physics experiment, which was soon diagrammed on a whiteboard: 

E-mini trades in Chicago propagate more than a thousand kilometers, at a substantial fraction of the speed of light, before slamming into data center “detectors” in suburban New Jersey. We could measure the latency by correlating the SPY order book response to E-mini price-changing trades. After obtaining data, sorting out FIX and ITCH, and building the order book — hassles all – we could see how fast trading was occurring.

To get a baseline, we first sifted data from the period around the end of April 2010 leading up to the Flash Crash. Those were among the now long-gone glory days of HFT. Huge volumes, huge volatility, and no sign whatsoever of anyone trading at a rate that could beat Spread, whose construction crews were racing to bore the tunnels for their fiber through the resistant Precambrian basement of the crystalline Appalachians.

 

Walking home that night, a Vampire Weekend song that rotated randomly into my earbuds seemed somehow apropos:

The pin-striped men of morning
Coming forward to dance
Forty million dollars
The kids don’t stand a chance

*