Making sense of sensor networks

Sensor Networks are Coming of Age

Sensor networks have been with us for years, but the combination of ubiquitous wireless networks, higher bandwidth, cheap storage, improved battery life and solar power are driving more and more applications towards data collection using sensors of one kind or another. These systems get very complex very quickly. Just consider:

• The sensors are usually distributed and heterogeneous
• Aggregate sensor data production rates are sky high (number of sensors times sample rate per sensor) particularly when combined with images and video
• The sensors can be fixed or mobile
• The people applications interested in the meaning of the sensor data can be fixed or mobile
• What is deemed important within the sensor data is fluid and constantly changing

How do you make sense of a massive stream of information, distributed across a large geography where the relationships between sensors and their surroundings can literally be changing minute to minute?

Why You Can’t Just Dump it in the Data Warehouse

Traditionally, the solution to problems with this much data has been to dump it into a data warehouse and let the data mining tools search for meaning among the gazillions of records. This is a proven way to index, apply rules and report on what happened, but it falls short in a variety of ways:

• It takes time to index and report, giving you an “after the fact” picture, not a current one
• Everything needs to be part of the master data model before it can be indexed and searched
• It’s notoriously hard to change what you are looking for, that is, to change the search and reporting rules

Data warehouses have their place and serve an essential function, but they don’t help you understand what’s happening right now. For example:

• A GPS sensor in a car could send data off to help city engineers understand traffic patterns and where to construct new lanes next year, but it’d be more satisfying if it could re-route you around the parade that’s blocking traffic right now.
• It’s good to know that a high radiation level sensor reading happened at the entry to the Holland tunnel an hour ago, but it’s a lot better to know that in real time when a truck is actually entering the tunnel. Similarly, it’s important to differentiate in real time between expected transportation of a medical device to a hospital (which can produce a positive radiation reading) and a radiation reading that is unexpected.
• Courier handheld scanners can feed a data warehouse with tracking information so you can look up the status of your shipment, but wouldn’t it be better if your mobile phone updated them on where you are and the package came to you? At work, at home, at your kid’s soccer match – no more “Sorry we missed you!” notes on your door. Science fiction? Nope, the day is coming.

Right Now: Understanding What, Where, When of Sensor Networks

What if sensors just spit out information and the infrastructure that was responsible for capturing and carrying that data could contextually filter and analyze the data to find the important real-time needles-in-the-haystack? What if that system was fully distributed in terms of geography, content and control so users of the system could define what they are interested in and many applications could find different meaning in the same data?

Today’s announcements make that vision a reality. Summarizing, Solace announced:

• The addition of a geospatial routing blade, making the high-volume processing of location-based information a core capability of our Unified Messaging Platform.
• A partnership with Thermo Fisher Scientific, the market leader in supplying sensors for a wide range of government and industrial uses, announcing that we have integrated Solace’s API with Thermo Fisher’s ViewPoint software. This means that any Thermo Fisher sensor that talks to ViewPoint is able to feed a Solace messaging backbone directly.
• And finally, the addition of Department of Homeland Security/Domestic Nuclear Detection Office as a new customer. Solace has been selected as part of a platform designed to enhance nuclear threat monitoring and response capabilities of local, state and federal emergency organizations.

Solace’s high throughput, content-aware messaging protocols effectively play matchmaker between the streams of data coming in from the sensors or other sources and the thousands, or even millions, of rules that define what to look for. It’s the size and scale of problem that is custom made for hardware middleware.

The Sky is the Limit

Today’s announcement focuses on emergency response at DHS/DNDO, but this same approach and architecture has virtually endless uses. For example:

• Environmental monitoring and study
• Command and control
• Manufacturing automation and controls
• RFID
• Smart grids
• Advanced surveillance and monitoring
• Home health monitoring
• Twitter or Facebook updates
• GPS data from cell phones

In all of these use cases, data starts out life as a stream of updates from hundreds or millions of endpoints flowing into a network. In each case, the difference between knowing what is happening now, and knowing what happened a while ago can be huge.