It’s a lot like having many little sources of latency in your trading system. Each weak link in the process keeps you from reaching optimal performance and profitability.

The best way to help people see what is wrong with their running form is to use video analysis and isolate in on alignment, landing points, hip rotation, arm swing, etc.

Finding and fixing the sources of latency in trading systems takes a keen eye and sophisticated tools, too. Today we announced a partnership with TS-Associates, a leading supplier of latency monitoring solutions. They’ve added support for Solace message routers in their TipOff  product, which is itself a hardware appliance dedicated to non-intrusive latency monitoring. In what may be the logical opposite of slow-motion video analysis, their solution studies end-to-end traffic in your trading environment and gives you details on which components in your system represent opportunities to cut out latency.

We find there are three groups that are hot to trot for this kind of latency monitoring right now.

• High-frequency trading speed freaks. These are the guys that are challenging the laws of physics in an attempt to be first in line for information and liquidity, and building their trading strategy around that advantage. They have been getting lots of attention from the press and politicians lately, but the “speed for speed’s sake” crowd is actually quite small. It’s an inherently difficult way to succeed over time, because for any given strategy only one algo can be first and it’s easy for latency to be leapfrogged out of the blue.
• All the other algo traders. This group still cares about speed, but generally aren’t aiming to eradicate every last microsecond from their system. They have unique trading strategies that are differentiated by something more sustainable than speed. They just want to be competitive, and above all, demand low variance in latency (jitter). This group represents the bulk of the buyers of low latency solutions right now.
• Complex animals like investment banks and major exchanges. These are the firms with trading systems consisting of countless moving parts. They need latency monitoring to get their arms around their systems and prioritize what part to optimize or upgrade next. Many of their processes incur milliseconds worth of latency, not microseconds, but they still need to be competitive with peers to stay competitive.

For any of these groups, the most challenging part of latency monitoring is that the problems vary by scenario and traffic pattern. For example, in a given system a risk check may be the long pole in the tent during a single message lab test, while one of the algo engines may be the bottleneck at typical data rates, and the 1 GigE network links become the choke point during periods of peak traffic.

A product like TS-Associates’ TipOff gives you neatly summarized data so you can understand where to put your attention to support each of these conditions in a production trading environment.

If you’re a runner with sore muscles or joints, get yourself a video camera and some good advice. If you’re a financial institution concerned about latency, check out TipOff. It’s fascinating stuff.

Last week Greg MacSweeney wrote a good article in Wall Street and Technology highlighting the back office as a new battleground for efficiency. After years of chasing zero latency for the front-office, the back office is comparatively archaic and badly in need of updating.

Greg predicts that 2010 will be the year the back office gets some love, and his analysis is spot on. Electronic trading has accelerated the rate of initiating trades so much that most mid- and back-office systems struggle to assess risk and execute trades before someone else gets there first. The mid and back office have to keep up with risk and make sure trade execution and settlement don’t experience bottlenecks or disruptions. But you can’t just apply front-office advances to the frumpier back office. If the front office is (mostly) about latency, the mid and back office are primarily about cost-effective, predictable and reliable behavior.

In financial services the cornerstone technology for making information flow is messaging middleware, and for front-to-back messaging, Solace’s answer is its Unified Messaging Platform (UMP). Solace’s UMP gives you one API that is best-of-breed for performance, reliability and operational simplicity across the front, middle and back office. It’s proven with big investment banks, financial information providers, hedge funds, high frequency trading shops and exchanges. I’m not talking about the future either, over half the companies we’re working with today look to us for mid and back office advantage in addition to speeding up the front office. Consolidating these traditionally segregated environments is changing the game in terms of value delivered for our customers.

With apologies to Justin Timberlake, we’re bringing sexy (to the) back (office)!

We look forward to discussing the announcements we made last week (DHS/DNDO as a customer, Thermo Fisher as a partner, and our new geospatial routing capability) and seeing what else is hot in the world of information exchange and critical incident preparedness.

In addition, we’ll be participating in a demonstration designed to show off how a range of technologies can help authorities manage a multi-faceted critical incident. In the scenario, an ammonia leak is detected in the Philadelphia Flyers’ arena shortly before game time, just as a severe weather front is moving into town. The scenario includes the routing of alerts and information as citizens are notified about the situation and given appropriate instructions depending on their location, hazmat teams are dispatched to the site of the ammonia leak, injured parties are sent to the best hospital for treatment, a tornado warning is issued, and more. The scenario includes practices and protocols such as geospatial routing, Common Alerting Protocol (CAP), Emergency Alert System (EAS), DM-OPEN, EDXL, Commercial Mobile Alerts System (CMAS) and EDXL Hospital Availability Exchange (HAVE). It should be an impressive display of the kind of information sharing being streamlined through the efforts of the National Information Exchange Model (NIEM).

Sensor Networks are Coming of Age

Sensor networks have been with us for years, but the combination of ubiquitous wireless networks, higher bandwidth, cheap storage, improved battery life and solar power are driving more and more applications towards data collection using sensors of one kind or another. These systems get very complex very quickly. Just consider:

• The sensors are usually distributed and heterogeneous
• Aggregate sensor data production rates are sky high (number of sensors times sample rate per sensor) particularly when combined with images and video
• The sensors can be fixed or mobile
• The people applications interested in the meaning of the sensor data can be fixed or mobile
• What is deemed important within the sensor data is fluid and constantly changing

How do you make sense of a massive stream of information, distributed across a large geography where the relationships between sensors and their surroundings can literally be changing minute to minute?

Why You Can’t Just Dump it in the Data Warehouse

Traditionally, the solution to problems with this much data has been to dump it into a data warehouse and let the data mining tools search for meaning among the gazillions of records. This is a proven way to index, apply rules and report on what happened, but it falls short in a variety of ways:

• It takes time to index and report, giving you an “after the fact” picture, not a current one
• Everything needs to be part of the master data model before it can be indexed and searched
• It’s notoriously hard to change what you are looking for, that is, to change the search and reporting rules

Data warehouses have their place and serve an essential function, but they don’t help you understand what’s happening right now. For example:

• A GPS sensor in a car could send data off to help city engineers understand traffic patterns and where to construct new lanes next year, but it’d be more satisfying if it could re-route you around the parade that’s blocking traffic right now.
• It’s good to know that a high radiation level sensor reading happened at the entry to the Holland tunnel an hour ago, but it’s a lot better to know that in real time when a truck is actually entering the tunnel. Similarly, it’s important to differentiate in real time between expected transportation of a medical device to a hospital (which can produce a positive radiation reading) and a radiation reading that is unexpected.
• Courier handheld scanners can feed a data warehouse with tracking information so you can look up the status of your shipment, but wouldn’t it be better if your mobile phone updated them on where you are and the package came to you? At work, at home, at your kid’s soccer match – no more “Sorry we missed you!” notes on your door. Science fiction? Nope, the day is coming.

Right Now: Understanding What, Where, When of Sensor Networks

What if sensors just spit out information and the infrastructure that was responsible for capturing and carrying that data could contextually filter and analyze the data to find the important real-time needles-in-the-haystack? What if that system was fully distributed in terms of geography, content and control so users of the system could define what they are interested in and many applications could find different meaning in the same data?

Today’s announcements make that vision a reality. Summarizing, Solace announced:

• The addition of a geospatial routing blade, making the high-volume processing of location-based information a core capability of our Unified Messaging Platform.
• A partnership with Thermo Fisher Scientific, the market leader in supplying sensors for a wide range of government and industrial uses, announcing that we have integrated Solace’s API with Thermo Fisher’s ViewPoint software. This means that any Thermo Fisher sensor that talks to ViewPoint is able to feed a Solace messaging backbone directly.
• And finally, the addition of Department of Homeland Security/Domestic Nuclear Detection Office as a new customer. Solace has been selected as part of a platform designed to enhance nuclear threat monitoring and response capabilities of local, state and federal emergency organizations.

Solace’s high throughput, content-aware messaging protocols effectively play matchmaker between the streams of data coming in from the sensors or other sources and the thousands, or even millions, of rules that define what to look for. It’s the size and scale of problem that is custom made for hardware middleware.

The Sky is the Limit

Today’s announcement focuses on emergency response at DHS/DNDO, but this same approach and architecture has virtually endless uses. For example:

• Environmental monitoring and study
• Command and control
• Manufacturing automation and controls
• RFID
• Smart grids
• Advanced surveillance and monitoring
• Home health monitoring
• Twitter or Facebook updates
• GPS data from cell phones

In all of these use cases, data starts out life as a stream of updates from hundreds or millions of endpoints flowing into a network. In each case, the difference between knowing what is happening now, and knowing what happened a while ago can be huge.

We are delighted to add BCC Group to our growing list of partners.

There may be some lessons to be learned from other areas of conservation that have been driving towards efficiency longer. There was a piece on NPR this morning about water consumption in which the US geological service released stats showing that per capita water consumption is down thirty percent over the past 30 years.

This dramatic reduction is primarily the result of improved efficiencies in industrial and farming techniques. For example, in the 1930s it took 200 tons of water to make 1 ton of steel. Today’s best steel plants use just 3-4 tons. Similarly, the evolution from flood irrigation to sprinkler irrigation and more recently to precision drip irrigation lets farmers produce more food with less water. Both of these moves to superior techniques and technologies were driven by the scarcity (and thus increasing cost) of water, along with a regulatory push in the 1980’s that established strict standards for waste water discharge. One of the least costly ways for companies to comply with those standards was to reduce the amount of water they used for their industrial and agricultural processes.

The inherent goal of business is profit, and public companies in particular have a responsibility to deliver maximum shareholder value. This means they can only afford to get really serious about “going green” when they are incented or required to do so, whether that’s in the form of market demand, increasing cost of resources, or regulation. That’s what happened with water, and it’s where we’re at with energy.

Exponential increases in market data rates, trade volumes and commerce conducted over the Internet are already causing datacenter sprawl that’s driving some insane energy requirements to power and cool the acres of servers they hold. So in addition to issues of scarcity and regulation, companies are already finding themselves highly incented to find technologies that can do more for their datacenter (whether that be in terms of storage, compute power or message routing) with a smaller “carbon footprint.”

As we continue to buy more gadgets to charge and eventually cars to plug in, will it be profitability pressures and regulations on industry that drives energy efficiency into the mainstream? I think the answer is clear.

Discovering the Joy of DM-OPEN

The agencies involved in the demo, and many technology vendors, worked closely over the month or so prior to the event to iron out challenges that cropped up in what should have been a straight-forward integration and compatibility exercise. The demonstration relied on the FEMA Disaster Management OPEN (DM-OPEN) communications tools. DM-OPEN is not a pub/sub system, nor is it really even message-oriented middleware. It’s a database that senders update and receivers poll for messages. Programs request all messages sent in the past X milliseconds, so even once a message is consumed, it will still be there the next time you poll. Because of indices on the database, you can’t send the same document twice – every new message needs a unique timestamp and sender ID.

There’s also a problem with how it handles time zones, such that applications need to poll by a timestamp in the message, instead of “sent time” on the server. This in particular led to additional implementation problems. There were inconsistencies in how participants were writing timestamps, which made matching events a challenge. For example, one vendor used a 12 hour format while the rest were looking for 24 hour format, and another’s messages were being completely left behind because of syntax problems.

In the end, lots of high touch communications and coordination paid off. Normally you don’t want your demo to be called a train wreck, but in this case, the blaring sirens and blinking lights meant the scenario had come off without a hitch.

Hard and cumbersome doesn’t cut it for emergency response

It’s one thing to get a bunch of smart people on the phone from the vendor and agency community twice a week to iron out issues—real-world emergency response situations clearly don’t afford such luxuries. In an emergency, dozens of organizations each need a filtered view of events that are, by definition, unexpected. This demo process placed a bright spotlight on some infrastructure changes that would make the deployment of emergency management systems much easier:

• Creation of a publish/subscribe message queue for communications
• System for guaranteed in-sequence, once and only once, delivery of messages from one to many
• Eliminate the need to embed timestamps or other arbitrary content in messages
• Get rid of schema validation that’s so strict it causes messages to be rejected for non-conformance.
• Enable filtration and routing of messages by geospatial coordinates and polygon inclusion

Geospatial routing is the key to information flow

This last point is very important, and it’s a new concept in message routing. In an emergency people have a literally “life and death” need for information about a specific area of interest. They can’t be bothered with topics and queues, nor with messages that don’t apply to them. It’s powerful to decouple senders and receivers so the only thing they need to share is the geospatial coordinates of latitude and longitude, or inclusion in a polygon. With geospatial routing senders don’t have to figure out which recipients they are sending alerts to, they can just “fire and forget” alerts with the confidence that the right systems and people will get them.

There are still two main types of application architectures in the world – request/reply where data is looked up as needed from a known location, and event-driven where new information is proactively sent to relevant systems in real time. Each has its uses, but there is no question that emergency response would be best served by a federated, event-driven information bus that can forward information between agencies and other organizations as events are unfolding.

First Forbes magazine did a cover story on The New Masters of Wall Street, highlighting the hedge funds, prop traders and big banks that are turning HFT into big business.

Then Traders Magazine did a related story about how the exchanges are responding in their Race to Zero Latency. Clearly the obsession with execution latency is being driven by the need to keep up with these ultra-fast consumers of liquidity.

And you know HFT is nearing its hype peak because last week the Daily Show did a special report on the subject.

Unlike Paris Hilton though, HFT is not making money because it is hot, it is hot because it is making money. Generating buckets of profits is one thing that never goes out of style.

How obsessive? Architects building these systems measure latency in microseconds, and the best applications exhibit just tens of microseconds of end-to-end latency. Shaving microseconds is like dropping weight before your prize fight weigh-in—whatever it takes, get it down.

To help these latency obsessed traders develop even faster trading systems, Solace has extended its Unified Messaging API to include a shared-memory transport based on inter-process communication (IPC). This capability lets two applications share information using Solace’s API with less than 700 nanoseconds of average latency in a shared memory environment. Yes, I said nano — billionths of a second. Remember the famous Tabb Report on The Value of a Millisecond? There are a million nanoseconds in a millisecond. 700 nanoseconds is a scant seven-tenths of a microsecond.

To be clear, IPC is a highly-specialized technique that only certain systems can leverage because it occurs within the confines of a single server. For example, when the components of a high-frequency trading system (feed handler, algo, risk assessment, order execution) have been consolidated onto a high-powered multi-core server within a collocation facility. Today these applications run on many machines and share data using low latency messaging (like Solace’s). Shared memory transport among applications running on a single server eliminates the few microseconds associated with network hops and additional time lags associated with copying memory around between applications. And since IPC is now available as part of the same API customers already use for ultra low latency and other kinds of messaging, applications get the speed they need without giving up the familiar API or the flexibility to redeploy in a networked scenario as needed.

As always, we’re not publishing some mysterious single number with no detail on what it means. A white paper describing the environment and parameters of the tests is available for download on our website so customers can dig into the facts and even reproduce the results using their own systems and data. In fact, we did all the testing a quad-core 3GHz Intel Xeon E5450 server because not everyone has the latest Intel Nehalem.

HFT architects have generally been exempt from corporate technology standards because the stakes are so high they can justify whatever makes them faster. With Solace, HFT no longer needs to be an exception. The same messaging API that is speeding up back office and front office networked trading can be used to speed up collocated HFT trading as well.

Solace signed a partnership with Sun shortly before the Oracle acquisition was announced in the spring. Sun’s technology assets and Solace’s high-performance JMS solution have generated interest with Sun customers since we began working together more than a year ago. Sun is a powerhouse in the data center of many of the country’s largest, most demanding companies, including most of the major banks. For our part, Solace is reshaping what has traditionally been software-based middleware into a hardware asset more like a network appliance than a server.

The combination of these two strengths is the basis of a new prototype appliance we demonstrated in the Sun booth at the SIBOS show in Hong Kong September 14-18. This new Sun financial network appliance combines the massive throughput and easy management of Solace’s messaging technology with Sun’s hardware and software to build a flexible solution for integration between back office banking systems and leading payment networks. Integration with payment systems is a notoriously complicated thing to get right, and the turnkey approach of this appliance can make connectivity faster, easier and more profitable for participants of banking inter-networks.

The appliance is based on the Sun Netra x4450 chassis, a powerful virtualization server that’s designed for telco-grade reliability in a low-power 4U form factor. It includes Solace’s JMS broker and middleware functionality with appliance-style turnkey administration. Finally, it includes Sun’s Secure Network Layer software as well as the Sun Integrator platform for easy integration of banking assets.

In related news, Solace announced a new version of our JMS protocol that’s used for this appliance, and is also available directly from Solace. It sets new standards for JMS performance with support for 100,000 persistent and 11 million non-persistent messages per second.

We look forward to a long and successful relationship with Sun (and eventually Oracle).