Good afternoon and welcome to the security seminar from Purdue University.
Our speaker today is Peter Bocci, a research scientist from NCSA at the University of Illinois, Urbana-Champaign.
He also has adjunct faculty positions with CS and ECE departments there.
His topic today is toward hazard-aware spaces, knowing when, where, and what hazards occur.
Peter, thank you.
So, welcome to the seminar, and let me first introduce what I'm going to be talking about today.
I would like to introduce you to our work environment, what kind of research and development themes we are working on, and kind of slowly go over the main focus of our research and development, which is multi-instrument measurement systems, what type of software and hardware we are using,
and then focus on the hazard-aware spaces project, where I'm going to try to define the problem, what are the technology components that we are using, and what are the research and development efforts that are related to integration of data across disparate sensing systems over time and space and variable.
And since this is a security seminar, I'll show the relevance to security research and try to kind of get you excited.
Why I am really here is really to introduce these problems to you and then show the significance of security to many of these applications.
I'll also mention some other applications, and I'll explain the term ex-informatics, which I have been using lately, and then summarize my talk.
I always like to start with giving credits and acknowledgement to all the funding agencies that fund our work, which is NSF, ONR, NIH, NARA.
We are really not very picky.
And we have about three to four, five students, full-time people, and then about eight to nine students.
At any time point, probably I can support about four students.
On this particular project, we had about four students.
So what are the research and development themes that we have been thinking about?
The first one is, clearly, when you look around and you see the novel technologies around you, the question comes, how can these advancements in sensor technology, computer science, and electrical and computer engineering help people?
Okay, the technology is around us, but can it help?
And then the question is, okay, where is the gap between the Moore's Law advancements and the use of these advancements by people?
Yes, you can have all kinds of cool stuff around you, but then how do you really use it?
Somebody has to take that effort and figure out things.
And then finally, because I used to work for machine vision companies and for other government contracting companies, the enormous amount of data that we are processing today, or acquiring and processing, and the tasks that we are doing and are very tedious and laborious,
they could be automated.
So really, my intention is to kind of free people from tedious and laborious tasks by designing multi-instrument measurement systems.
There is no single instrument that can solve certain difficult problems, but if you take many, many instruments and you put them together, you might be able to solve very difficult problems.
So as I mentioned, our goal is to understand multi-instrument measurement systems, and the focus is to develop theoretical models and perform experimental understanding of multi-instrument measurement systems that deal with multidimensional, multivariate data,
and include multispectral cameras, multimodal imaging instruments, sensor networks, wireless communication, data acquisition, fusion, analysis, modeling, and visualization components, whatever you can think of.
So it's a very broad spectrum of domains that we are trying to cover and go from sensor instruments through data acquisition, data integration, analysis, and synthesis.
And clearly, there are many, many applications domains.
I just picked three that we published and worked in, which is the biomedical or bioinformatics and medicine domain.
And we have a lot of projects in the GIS, geographic information systems domain, especially in the agriculture domain, because we are like Purdue, we are in the Midwest, and we work with the agriculture department, hydrology, water quality, structural and earthquake engineering,
environmental engineering people.
And then we are looking at these novel sensors and then trying to build smart spaces.
Clearly, military is interested, machine vision is one application, plant biology, and healthcare.
Our little imaging lab that we have has several cameras.
We have high-perspective cameras.
If I would ask you what's a high-perspective camera, really, if you take a regular camera and you take a picture, you will get, let's say RGB, you get three values that cover the wavelength range from 400 to 700 nanometers.
But if you have a high-perspective camera, you can have hundreds of values.
So that means you might have much better discrimination when you are looking at different shades of color.
We also have multi-spectrum camera that NASA is using, for example, when they deploy space shuttle or rovers, and they want to explore environment.
They have, let's say, red, green, blue, and infrared, and everything comes together.
We have thermal infrared camera that detects heat, and then we have wireless and wired cameras so we can build systems with these.
We also have a robot with robotic arm, which we are using for deploying sensors.
And then when we want to control robots, we use, for example, gesture, and then we use orientation sensors to remotely control deployment of sensors.
And the sensors that we are currently using are the Crossbow Smart MEM sensors, and I will show them a little bit later.
We also have audio, so I can, the same way as I have the microphone now, I can say move ahead or give commands, and the robot is going to perform them.
And then we are looking at RFID tags and readers, and that will be part of our environment as well.
In terms of publication and software, we try to document our work so we can disseminate our knowledge, and there is a software tool called Image2Knowledge, and you can find the manual on our website, and also there are many, many publications about our solutions,
and you can see the variety of pictures that I put here.
Those are the types of data that we process from very, very different domains, and at the end of the presentation, I will show you what is common in all these domains.
So just to give you a quick list of examples of problems and solutions and applications that we have been dealing with, clearly, let's say crop yield prediction using hyperspectral data, instead of sending graduate students to collect measurements in the ground,
you can have a little Cessna flying over your farm field, and then you collect hyperspectral data, and you can predict exactly what's the soil conductivity, what are the other measurements, and then you can predict your crop yield.
Spectral scene modeling, for example, you want to fly tomorrow, and you want to predict that scene from a certain view angle.
Can you reconstruct that scene based on terrain map and elevation maps and so forth?
And possibly BRDF stands for Bidirectional Reflectance Texture Function.
Historical map analysis, again, you are doing a restoration project.
You want to restore some area close to a river.
You want to go and recover maps from 1900.
You scan them.
You want to recover the information by isocontour so you can restore that area.
And then we deal with integration modeling of large-size remote sensing data for hydrology community.
We build a lot of decision support systems.
And today I'm going to primarily talk about the hazard-aware space and the robot control using arm gestures.
A little bit about sensor registration and fusion.
And then there is another set of applications that deal just with bioinformatics issue and medical issues.
And I will not mention them today.
We also, since we are part of the university, we are trying to educate students, like what you are here, we are more or less building tools for the students to use and then figure out things like, okay, how to do image operations, how to understand sensors and similar.
So let me now introduce the problem, the hazard-aware spaces.
And I put there a simple question.
If you are in a smart room, what you really want to know, it's a basic question, where, when, and what type of hazard occurred.
So you are sitting in your office, you are studying.
These are the basic questions that you want to know if you are building such a system.
So if you think about it, there is a smart room, let's say, in this audience, or in this room where we have sensors that report, let's say, temperature.
And then the temperature might rise in this corner, and then there might be a camera that's looking and then checking whether it's really hazard.
Maybe that was just a false alarm.
Then you analyze what's the type of fire, whether it's a burning wall, or it's the carpet, or it's the desk, and so forth.
And then you are trying to trigger some event.
For example, you can call 911.
Or you can call a robot and extinguish or contain that hazard.
So you are really trying to put together multiple sensing devices, gather the information, figure out where the sensors are, at what time the measurement was taken, and what kind of hazard is occurring.
So if I want to describe the problem, I would say, well, given wired and wireless multispectral cameras and other point sensors, assess hazards in an indoor environment.
And my inputs are the wired multispectral sensors, like the thermal infrared camera, hyperspectral camera, visible and multispectral camera inputs.
And then I have these point sensors that have onboard processing capabilities, and they can measure temperature, acceleration, luminance, all kinds of variables.
And my system requirements are, well, I want to adaptively position these sensors in space and time, possibly switch between sensing temperature and luminance and other variables.
Then I want to have minimum wireless loss in the space, so there is an issue with transmission and reception.
And then I want to optimally integrate all the data across disparate sensing systems over time and space, and provide accurate and fast calibration.
If I have a reading 10, I don't know whether it's temperature, 70 degrees Fahrenheit or 100 degrees Fahrenheit.
I have to calibrate it.
And then I sometimes might get false alarm, or from a security viewpoint there might be a malicious sensor that's reporting false alarm.
I have to have confirmation mechanism and hazard understanding, and then somehow I have to alert humans.
Usually in this kind of scenario there is always human involved, so I have to have some kind of interface to a human, and then possibly develop an application that has the hazard containment component.
So the motivation for doing this work is really, from a research perspective, is to figure out certain proactive surveillance system, similar to a thermostat.
So you don't really know that your thermostat is working at home, but it is switching on and off your heating.
There are bandwidth requirement reduction issues.
So for example, in this room, if I install here 20 cameras, the bandwidth that I need for those 20 cameras is going to be huge.
But if I'm just sensing with these point sensors, it's very low bandwidth, and I'll trigger the cameras only when I know that there is some kind of hazard.
So the bandwidth, it's very efficient use of the bandwidth.
I also want to introduce spatially adaptive sensing.
There might be sensor failure.
It might be too expensive to put too many sensors in this room, so I want to be able to reshuffle the sensors in the room.
And then I would like to understand the hazard.
So when I call 911, I'll say, okay, it's in this room, and it's the furniture that's burning.
Please bring appropriate extinguisher.
Now, from a technical viewpoint, the challenges are, when you are building a system, you have to worry about the sensor selection, and what's the ruggedness, weight, capabilities of these sensors.
There are some power, memory, and CPU constraints.
In terms of wireless communication, you always are concerned about limited range, delays in transmission, packet losses.
How do you deploy sensors?
What if we would be deploying sensors in the radioactive, in a room that has radioactive material?
We don't want to go there.
We want to have some remote control over the deployment.
Then, how should I deploy the sensors?
Should they form a grid?
Should they be in a straight line?
Should they be close to a wall?
What's the optimal sensor layout?
Then I have to worry about the sensor and camera calibration, camera control.
I'm switching, I have multiple cameras, so I want to switch which camera is currently broadcasting.
I want to have the capability to recognize and detect hazardous events, and then possibly confirm, and then propagate the alarm.
So maybe I alarm a security guard, but he is sleeping.
Am I going to wait for half an hour and then propagate the alarm to his superior?
Or how is that mechanism going to work?
So this is what we developed so far.
This is a prototype.
We started with the sensor deployment aspect, which is left in the middle of the slide.
And then we developed techniques for deploying these MICA sensors that I will describe in the next few slides.
Then we started to worry about synchronization of cameras and sensors.
How do you synchronize all the measurements?
Since we deployed the sensors, we want to know where they landed.
So we applied multiple algorithms to localize the sensors.
Now we start to acquire data.
We want to know what's that measurement?
What's the interpretation of the measurement?
Is really raw value 10 a hazard, or it's just a regular ambient temperature?
So we have to calibrate spectral cameras and sensors.
Then when I turn off the light in this room, you don't want to be sending visible spectrum camera to the operator because he wouldn't see anything.
You want to turn on your thermal infrared camera, and that makes sense.
So we have this proactive camera control, which really works without us really knowing about it.
And then sometimes you want to be doing regular checks, and you want to go and send the robot and walk into each room and say, well, find and scan the room and tell me if there is any hazard.
So there is this hazard detection algorithm that we have in place.
And then we are not really human computer interface people, but we are trying to figure out how would you build a friendly user interface so the operator immediately sees what's the temperature, whether the light is on or off, he can switch between cameras,
he can look at the sensor readings, and then eventually when he sees a hazard, the hazard will be highlighted in his view so he can make a decision.
And then there is a component that deals with hazard understanding and confirmation.
I'll talk about it a little bit later.
And then eventually hazard containment, which I'm sure will amuse you because we just built it in the lab for demonstration purposes.
So when you think about the whole system, it's a combination of research and development.
When you think about development, you think about deploying the point sensors using robot control.
But when you think about the basic research, you really have to figure out how to control the robot remotely and how to adaptively set up your application.
How do you deploy the sensors?
When it comes to calibrating cameras and point sensors, again, there is a research component.
What is my accuracy?
Can I quantify the accuracy?
And then, can I do it in real time?
Speed is very important.
Then it comes to controlling cameras proactively.
It's very important for bandwidth control and information selection.
What type of camera am I turning on?
When it comes to the hazard detection, again, there are many, many issues how to understand the hazard.
By just looking at the flame, can you tell what is burning?
The alert operator, again, there are research issues with the human-computer interface.
And when it comes to hazard confirmation and containment, you want to eliminate small hazards.
In my opinion, these are new applications for robotics because clearly you cannot deploy robots in any scenario.
It has to be well-defined.
You might know the CAD model of the building or a floor plan and so forth.
But it's a new...
It's a research issue how to do that successfully and reliably.
So, here's the list of technology components that we put together.
So, we can deploy point sensors in smart mode using robot control using either human-computer interface, keyboard, mouse, or using speech or using gesture.
And then we can...
we have video feedback from the robot so we actually can verify and possibly track a few things in the room.
Then we have the component that deals with calibration, time, space, and measurement, all the spectral cameras and point sensors.
Then we have a component that switches the cameras, which we call proactive camera control using point sensors between visible and thermal information.
Then there is a component that deals with hazard analysis and human alert which is really very much related to emergency management and response.
And finally is the hazard confirmation and elimination using robot control which means really we have to go back, deploy more sensors, and we will attempt to contain hazards.
Related work...
clearly the challenge here is that the use of technology components is scattered all over many fields.
The wireless sensing devices are frequently used in smart spaces, ubiquitous and proactive computing and situation awareness applications.
Robots are used in industrial automation applications, autonomous modes of exploratory operations and others.
And the spectral cameras are used in detection and recognition systems where 2D and potentially 3D spatial and spectral characteristics are very informative.
But these are very different communities many times.
And when you search for publications, so you don't reinvent the wheel, really publications are scattered in many domains addressing individual problems of antenna design.
Many people are working on can I really adaptively change the antenna of my point sensor so the transmission and receptions have much better efficiency.
Sensing, tiny operating system, communication protocol, synchronization schemes, wireless programming, quality of service, security, reliability.
You can find a journal about each of these issues, I would say.
And the same applies to robot control, tracking, automation of repetitive tasks, autonomous operation of these robots.
And then if you go to the remote sensing community, let's say you will find many of the camera calibration tasks, image understanding, real-time video processing, automated recognition, classification, compression, security, distributed computing, you can think of it.
So you can see that it's a very broad spectrum of problems and related work.
So our work really embraces a very broad set of individual problems and proposes and leverages from data integration across disparate sensing systems over time, space, and variable to improve system performance.
And eventually reveals advantages and limitations of novel technologies.
And then finally the discriminator is that we deal with real-time monitoring applications.
In comparison with past work, most of the past successful applications are not real-time.
For example, the environmental wildlife applications.
There are applications in Maine, they use the sensors for monitoring some species like birds.
Let's say in California they use it for monitoring temperature and moisture of trees at different height.
But these are very static applications.
For some applications the sensors even fail.
For example, when you measure earthquake, clearly the sensor is not capable of being idle for let's say 20 or I don't know, let's say 2 years.
And then suddenly there is an earthquake and in that fraction of a second you want to have sampling frequency once in hundreds of a second.
That sensor is not capable of doing that.
So for some applications these sensors are not even appropriate.
So let me show you the first component when we deploy point sensors using robot control.
It is a quick introduction to these sensors.
They are called micro-electromechanical systems based on this technology.
We call them smart because they have some computing and communication capability and there is a sensor board that has some sensing capability and we run an operating system called TinyOS and it was all developed by Berkeley.
The hardware consists of the motherboard that has a 4 MHz processor and it has a radio board so it can communicate and then the plug-in sensor boards have different functionality.
The one that we use have light and temperature right now but we have others that have microphones and buzzers, they have GPS and so forth.
The challenge is that there is always a battery attached to these sensors so you have to worry about energy and the efficiency of your processing.
And then the interface board is attached to your PC so you can program these little sensors and also when the wireless sensors are communicating with your laptop they go through the interface board.
And here is a demo of how we deploy the sensors, very simple one, more or less we load the platform of the robot with the sensors and then we just remotely control the robot to move it to the desired location and then the robotic arm is going to deploy the sensor.
This is autonomous or it's autonomous?
We control the robot deploy.
So the robot knows exactly the location of the sensors and we just say, okay, move to this location and deploy.
Now if we pre-program and we say, okay, now you constructed a brand new building and I want to deploy sensors in each room at this location, then it could be autonomous.
Okay, so you can also see that as I was deploying these sensors we had to solder the antenna to each sensor because it didn't come with it.
And then we did many experiments that really explored what's the dependency of the orientation of the antenna and the shape and so forth.
And again, these are electromagnetic issues.
But our prime focus was to develop multiple interfaces and here I'm showing our client server architecture of the robot control system where we have the keystrokes, clicks, and mouse which are processed by one PC which is really one interface to control the robot.
Then we have the speech as long as I have a wireless microphone, I say move ahead and then the robot is going to execute that.
And then we have the gesture recognition.
The gesture based robot control is probably the most interesting and it originated from Navy.
And it had the following problem.
In these days, Navy aircraft carrier decks have UAVs, unmanned air vehicles.
When you go to the airport and you see people who are called yellow jackets or flight directors and they show move ahead and pivot left and so forth.
The question is can you build a system where you don't change anything in the current procedure but the UAVs can be controlled the same way as these aircraft that have pilots in cockpits.
The challenge is that in these kinds of environments you have limited line of sight.
There is a lot of interference, electromagnetic interference, acoustic noise, so you cannot use speech.
And there are many emitters and these people who are working on Navy aircraft decks are usually there 14 to 16 hours, so fatigue is a big issue.
And they are exposed to temperatures between minus 10 and 140 Fahrenheit.
So really the technology that you build has to be very robust.
So what we decided to do is a simple demo in our lab where we mounted some sensors on a human and then collected data did the recognition of the gestures and then converted the gesture command into the instruction set of the robot and the robot was performing the command.
So the motivation and challenge is one more time.
The motivation is to allow the UAVs to mix seamlessly with the existing traffic on an aircraft carrier deck.
And the challenge is what types of sensors you should choose whether active or passive.
How do you attach sensors if you choose active?
How do you acquire the data and what's the repeatability?
If I'm very very tired my move ahead is going to look very different than if I'm fresh.
What if I'm female as opposed to male?
Do I have different gestures?
Then clearly there are some automatic classification issues.
How do you classify what's the reliability of that?
Everything has to be real time.
That's another challenge.
And then eventually you have to translate the hand gesture sequence into a sequence of autonomous vehicle commands and then execute and then get some feedback.
So those are also challenges when it comes to designing such a system.
In terms of related work, clearly other approaches have been proposed in the past.
There are vision based systems using passive sensors.
If you don't want to change anything in the Navy scenario you just put some cameras there and then you compute stereo and there is either single or multiple cameras.
You acquire the video stream and then you map the gestures into temporal signatures and then you try to do the interpretation of that.
The challenges are that in a harsh environment the flight director, there is a different orientation of the flight director, the outdoor illumination is changing all the time and there are possible occlusions.
So the vision approach is not going to work too well.
We looked at the devices with active sensors that are devices like gloves with bend sensors or miniaturized accelerometers and again the constraints are that it cannot be very heavy because these people are there for 14 hours and it has to be rugged and so forth.
So eventually we decided to go with small sensors that report Euler angles, which is yaw, pitch and roll.
And then there is a mapping, unique mapping between Euler angles and gesture motions.
And it was satisfying all the application requirements.
And then we published a paper that more or less analyzes what's the repeatability so the signs really denote that we can discriminate gestures that are similar from those that are dissimilar.
And then eventually the classification can work real time with some small delay here.
You can see here I'm showing just the pitch signal that comes from four different sensors and then by analyzing we can immediately say it's turned to left and command.
And here's an example that actually we analyzed later on.
What if I'm using microphone and I'm using gestures and I'm saying stop while I'm showing move ahead.
What if there are conflicting commands?
Now, the other issue is, okay, so what if there are consistent commands?
Which commands should be executed?
But if there are conflicting for example, from the security viewpoint, you might have an intruder who's going to intercept the signal and he will be saying move ahead while the aircraft is close to the edge of the carrier, the ship, and it's going to fall off the board.
So how do you potentially deal with this?
So again, you design a system that can deal with that.
And then you can read about our publication.
I have a list of publications.
Okay, so now we deploy the sensors.
We want to know when hazards occur.
So it's really a synchronization problem.
So the problem statement is, given a set of wired and wireless instruments and sensors, synchronize all measurements.
Clearly, there is related work.
There are network time protocols, digital time synchronization protocols, security protocols for sensor networks, a lot of work.
There are journal papers about this, tons of publications.
What we've done is, we said, look, our wired cameras, they can use the atomic clock and they use the network time protocol.
Everything is connected via the internet.
But the challenge is, when you have these point sensors, and they have to wirelessly communicate with a PC, how do you synchronize those?
So we view them as stand-alone PCs, pretty much.
And what we decided to do is that at the start-up, the PC, the laptop, is going to send the current time to each of the mica, these point sensors.
And then the mica sensor, because it has a counter, will update its timestamp every x milliseconds, let's say 10 milliseconds.
And then each reading, so after I measure temperature, I'm going to be sending a packet that includes the reading and the counter back to the laptop, which is the timestamp.
And then the laptop is going to look at the difference between when the sensor reading is acquired and the time when the reading is received on the PC.
And if it's larger than some threshold, in our case it was 350 milliseconds, then the PC and the mica sensors will resynchronize time again.
So that was the procedure.
And that worked pretty well.
So we synchronized them and we asked the question, where did these sensors land?
So where did they land?
Again, our challenge here is we want to determine 3D location of the deployed wireless mica sensors and possibly other objects in the scene, assuming that the camera locations are known, they are static.
Previous work, again, can be split into two different areas.
One is the 3D object recognition community, the computer vision community, machine vision, signal image processing communities, and they use, let's say, shape from cues like stereo motion shading and they get the depth map.
And there is a different community that deals with these wireless sensors and they talk about location sensing, where they use radio or ultrasound time or flight lateration or signal strength analysis.
And there are systems like MIT Cricut Indoor Location System and the AT&T Bat Ultrasonic Location System.
They are just trying to solve this problem.
The problem is very difficult on its own.
So what we did, we proposed a fusion.
So again, we know that there are different communities, different solutions to this problem.
But we said, well, what if we take the stereo approach, the vision-based approach, we compute the depth.
Then we use the localization approach.
Again, we compute where the sensors are approximately located.
Then we perform data fusion and then we get much better location of these sensors and all the objects in the scene.
So that was the basic idea.
The way how the fusion works is that you look at the stereo first and you say, okay, many of you probably know that if you compute stereo, as you go away from the camera, the uncertainty is going to increase.
So clearly, the objects that are close to you, they will be more accurately predicted than the ones that are far away.
So we computed something called depth uncertainty and when it came to the sensor localization, for those of you who don't know how this works, clearly these sensors have acoustic and radio transmitters.
There is a buzzer and the sequence goes like this.
First, the base station is going to send a message to all the micas and say, we are going to be doing this localization.
Then sensor C and B, in this case, would send acoustic and radio ranging transmissions everywhere.
And then the sensor A would receive the radio ranging transmission and after a while it would receive the acoustic ranging transmission because acoustic waves are slower.
And it would measure the difference and report back to the base station the difference, which is really useful for estimating the Euclidean distance.
The challenge was, if you look at the upper right corner, when we used our sensors, the pink line is really what we expected.
We had sensors apart, let's say, one foot, two feet, three feet, and so forth.
And then when we measured the blue curve, the difference was pretty significant.
I think the error was about two meters.
So we tried to compensate it, but it was not very satisfactory.
Then we realized that what we have to do is use much more powerful buzzer.
And we borrow some sensors, and we get much better results.
So here, the distribution of the error as a function of distance between two adjacent sensors shows that we get better results.
However, there is a physical limitation.
If you take two sensors because the counters have finite sampling frequency, you cannot get better than 30 centimeter accuracy.
So there is always some uncertainty.
So by understanding the uncertainty issues, we build a fusion mechanism which really is shown in the middle.
I don't have the pointer, but more or less the curve shows the pink or purple graph shows that the uncertainty for the stereo is going up as you go away from the camera, while the localization was pretty much stable for certain range.
And remember, this is only single hop transmission.
And here on the right, you see that the measurements that were originally obtained by the localization algorithm were improved after fusion by a certain percentage.
I think it was by 27 percent.
The same way we could take the depth estimation from the stereo algorithm and improve it with the point measurements obtained from the localization, and that was about 98 percent.
But if you can find the threshold which is shown there, there are two regions.
I think I can use my mouse here.
So there is a region where you should be using just the stereo, and there is a region where you should be using the localization, and that way you get the best result.
We even built some theoretical models of these fusion parameters so we can predict what should be the fusion threshold for multiple setups.
And that really is based on the uncertainties of these different methods.
On the left side you see if we fix the depth map uncertainty, how is the surface going to intersect with the localization that also varies.
And then on the right side is the other situation.
Okay, so we know where the sensors landed, we synchronized them, now we want to do some real hazard detection but we have to calibrate the measurements.
So the problem is convert point sensors and camera raw values into engineering units to enable hazard detection and recognition.
For example, degrees of Celsius or Fahrenheit or Kelvin for temperature measurements.
And the previous work, again, it's scattered in multiple domains.
It's in the remote sensing community.
They use it for calibration.
Robotics, again, you want to build autonomous vehicles so you want to calibrate your inputs and analyze calibrated imagery.
In astronomy, they are very much concerned about calibration because they are looking at the sky and trying to understand the stars.
Military, clearly.
So, our primary approach was to do a black body or temperature gauge calibration and the schema that we decided to use is that we either calibrate the point sensors or we calibrate the thermal infrared camera and then we use it for calibrating the other sensing modality.
So, you would say, well, the sensors can be the calibration formula.
This is what comes from the manufacturer.
And I can show you that here on the right side, if you apply the calibration formula from the manufacturer, you will get readings that have very widespread.
So, still after you use that formula, you get very widespread and you have to calibrate the sensors.
Now, imagine if you are deploying millions of these sensors, you have to take each of them, put it on the interface board, program it, calibrate it, once you are finished with the calibration, take another one and do the same.
Millions of them.
So, what we proposed is, well, if you take the thermal infrared camera and you calibrate that one, which we've done in the lab, then you can take a picture and you have millions of them, or I don't know how many, would fit into your field of view, and in one snapshot,
you can calibrate all of them.
So, I think that might have a significant application for the sensors.
On the other side, if you say, okay, now I calibrated all my sensors , I want to calibrate my camera, you can, just a second, you can more or less put the sensors in your field of view , and repetitively, you can calibrate your thermal infrared picture.
Yes, please.
Before you deploy them, you have to program them.
So , why don't you calibrate them while you are programming?
Calibration, okay, so how would you calibrate them?
Like if it's a temperature sensor, you can have like four or five temperature points , and then you can calibrate when you are programming .
Right, but remember that in these days, they are working on the wireless programming interface, so you don't have to put them on each.
Okay, so there are two things.
If you put them on the interface board and you are programming, you are saying that at the same time, I'm going to vary the ambient temperature, and collect maybe four points, the least square feet, and then kind of calibrate them.
So that's one approach.
But still, you don't want to do it for a million sensors.
And in our experience, we really want to spend the least amount of time reprogramming them.
And the calibration software that you upload might be very different from the operational software that you want to upload.
So that was the motivation for that.
I have not used the wireless programming capability for these sensors.
But I know that Berkeley has been working on that quite a bit.
And whether it's a use...
how much you can use it, I don't know.
But we are thinking about these two different approaches, how to calibrate whether sensors or the cameras.
There are many, many, many basic questions when it comes to optimal sensor network design.
And I'm just listing some of them.
Is there an optimal spatial arrangement of sensor networks?
We look at the linear arrangement, circle arrangement, and so forth.
What is the impact of other wireless devices on sensor networks?
We did the experiments.
What if you have your cell phone on and suddenly all the experiments would report other...
would fail.
What is the dependency of wireless loss on the number of active sensors?
We actually were probably...
nobody really wanted to hear this, but we actually measured that as you are adding more and more sensors in your environment, you will incur more and more loss.
But people didn't want to hear that.
What is the most effective mechanism to receive information from the wireless sensor network?
And the complexity is pretty large because we played with about four different communication protocols just to figure out should we buffer more?
Should we send immediately?
Sending is expensive.
It's going to drain the battery.
And then where is the trade-off?
How do you synchronize data acquisition?
I already addressed that issue.
How to maximize information content of received sensor readings?
Well, I might not want to send the timestamp, but then what's the value of that measurement if I don't know when it was taken?
How often should one sample?
What are the trade-offs?
Sampling rate versus wireless loss?
How to preserve energy?
So there are plenty of research topics.
And I'll just show one of them that we kind of looked at, and that's the sampling.
Clearly, if you increase sampling, you get more and more measurements, but suddenly you have more traffic.
If you have more traffic, you will start to lose packets, and at some point, no matter how much you increase the sampling, you will not be getting back the samples because there is so much traffic and so much information loss.
So again, what's the trade-off?
How do you design such a system?
And again, on the left side, I just listed the wireless challenges.
Okay, so now let's suppose we set up the environment, and we want to go to an operational mode of the system.
And one example is the camera control.
You want to perform continuous low bandwidth data acquisition using microsensors to monitor fire hazards and then trigger high bandwidth data acquisition using thermal infrared or visible spectrum cameras to recognize types of fire hazards.
And again, the past work is primarily in the surveillance applications, and Berkeley has done some experiments with air conditioning and lighting systems.
Our proposed approach was just really to focus on the bandwidth control and then figure out the proactive camera selection.
And the logic is very, very simple.
More or less, if the light is on, then turn on the visible camera.
If the light is off, then turn on the thermal infrared camera.
And you can see the interface here more as we just want to report to the operator what's the temperature, what's the light, and then we would clearly show just one of the video streams.
Hazard analysis.
Okay, so now we want to detect the hazard.
And the problem, you know, there are many, many variations of this problem.
You can go and scan indoor environments for hazards on a regular basis.
You can acquire video and then search for hazards.
Excuse me.
And then if hazards were found, then identify the source.
And our approach was to take the color and the temperature-based hazard detection and potentially use the spectral image analysis to understand hazards.
So here is a video of the algorithm that we implemented.
So if a robot walks into a room and we had some candles on the whiteboard, and as it goes it is color-based detection of the places that form a hazard.
And then it would report to the operator, here is the location.
And not only that, but if you want to deploy a robot and while the hazard is small, you might want to contain it.
Excuse me.
You want to know the orientation and the location and so forth.
The human alert was another issue because when the operator is not paying attention, he might miss the information that there is a hazard.
So how do you redirect his attention to the hazardous situation?
And right now we just created a simple overlay so we can show that there is some kind of hazard.
Potentially there can be a beep or something like that.
And then eventually the whole system would have a little bit more complicated human alert logic where when the temperature is higher than some threshold and the light is on, then turn on the visible camera, overlay the hazard, the region, and then maybe beep so the human is alerted.
And if the temperature is larger than threshold but the light is off, then it would show the thermal IR camera output and the overlay on that.
We are very interested in hazard understanding.
We've done a lot of work in the past with hyperspectral camera and the basic question is if you take a picture with the Canon camera and the Sony camera, which one , what is the true red color?
Because when I'm looking at fire and I'm trying to understand what is burning, I'm really basing my conclusion is based on the color.
But do I know what's really true red, what's really true yellow and so forth?
And I bet you that if you have two cameras, they will give you different values.
So we did some work.
We used the hyperspectral camera to establish ground truth and then we can use the ground truth for calibrating single RGB cameras.
We also went back to Maxwell equations and actually the work that we did for a scene modeling project, which is shown on the left side, if you can get a very good representation of multiple fires and you can build a database then you can even predict how that fire is going to look and eventually classify the fire based on the appearance.
And what is shown on the left side, we build a, this is called a bi -directional texture function and a database of that for multiple clutters, scene clutters.
And then from measured images we can predict images once we know the layout.
But unfortunately we didn't finish this part because it's really, really very hard problem but we want to be able to discriminate a fire of a sofa which is hazardous as opposed to a fireplace where you might have a fire and it's not a hazard .
Hazard confirmation , so again the motivation is to confirm hazard when MICA sensors report inconsistent values.
They might be malfunctioning or there might be malicious sensors.
Hazard measurements might be too sparse and it is desirable to increase spatial density of sensor readings.
And the hazard elimination might be desirable if large scale damage could be prevented or minimized by expedient hazard containment.
So while the hazard is small it might be much easier to contain.
And our approach was to deploy more sensors using the robot and then kind of try to demonstrate a new application of robotics for hazard prevention and small hazard containment.
And again I'm showing this movie just for amusement pretty much because I brought a hair dryer from home.
We put it on the platform and we used the robot to move closer to the hazard.
The robotic arm is going to turn on the hair dryer and extinguish the fire.
I'm saying that it's for your amusement because many times I'm trying to demonstrate to our funding agency that there is a potential.
I'm not saying that this is the way how it would operate in a real case.
But you can see the concept.
So that's how it works.
Okay.
If you are interested in learning more about this kind of work, we published several papers on the remote robot control, the fusion issue we addressed in one paper, the repeatability assessment of the gesture recognition, and the whole classification algorithm.
The localization and stereo fusion, one of my master students just graduated so he has a master thesis about it and we have some papers.
The measurement calibration, again we have a set of papers and one student graduated about two years ago.
And then the system design issues, they are summarized in a technical board because I'm trying to kind of summarize our work so other people can learn.
There are also publications about the hyperspectral image analysis, but again we haven't figured out fully how to do the fire classification.
There are some journal papers that are related to our past work.
Relevance to security research.
So you are all coming to the security seminar and what I want to convey to you is that building hazard aware spaces requires an interdisciplinary team and expertise as demonstrated in the presentation.
And security is critical to protect the information and to have safe operation of hazard aware spaces.
I just want to emphasize that these wireless sensor networks are used for emergency and life critical applications so we want to make sure that nobody is going to mess up the system.
And the question is how to build into a system data confidentiality, data authentication, and data freshness in this resource constrained environment.
From my perspective, from the application viewpoint, security is really needed for the wireless robot control, the fusion of multiple robot controls, the communication of the wireless sensor networks, the sensor programming, if it starts to be wireless,
we have to be concerned about it.
The camera control, we don't want the intruder to use the camera for doing something else that it's supposed to do.
And the video transmission.
So, in general, I would say the need is to build some kind of security services for multi instrument measurement systems.
That's what would be really useful for the wireless sensor networks, for the data gathering aspect, for multiple instruments, and then for the fusion for the multi instrument sensor data integration.
Because remember that in many of the applications, eventually when you graduate and you get a job, most of the top three application requirements are accuracy, reliability, and security.
I also want to say that there are applications of my interest that are some ongoing wireless sensor network projects funded by multiple agencies.
And the red ones, I highlighted because I'm involved in those.
So, I will just mention that NSF has spent a lot of money on funding these sensor projects.
For example, Cleaner is the environmental engineering project that deals with deploying sensors and measuring some environmental variables so they can predict, let's say, the waste management and the nutrient nitrate, or nutrient nitrate nitrogen levels in rivers.
Kuhasi is another project which is more about hydrology.
And again it's concerned about data acquisition and using remote sensing data and ground data and putting that together.
Shield is a very interesting project and I'm on the advisory board of that project where they are trying to embed sensors into bulldozers, tractors and so forth.
It's led by Caterpillar.
And why?
Because it's so expensive to repair these big
vehicles so it's much cheaper for them to embed these sensors and the sensors will report when there is material fatigue.
So that's another one.
And then the healthcare monitoring, clearly for the healthcare at home, hospitals, battlefields and so forth.
I just want to mention that what I am trying to show is that the integration is really transferable to other application domains.
The multi-instrument measurement systems are present in many engineering applications and there are application-specific instrument sensors but the information processing issues are similar.
And therefore I kind of started to use the word ex-informatics because it refers to the process of processing the data.
And if you are interested there is a book that is just coming out this month, Hydroinformatics.
I wrote about 250 pages.
The whole book is about 650 pages that deals just with the informatics issues in the hydrology.
So if you are interested, there are bioinformatics books and I published one book chapter here which is dealing again with informatics issues.
But as I am saying, it's very transferable.
And then guess what could be on this slide?
It's not just hydroinformatics and bioinformatics, it could be also sensor informatics.
To summarize the talk, I really wanted to just overview the research and development of these complex systems, introduce you to hazard-aware spaces, and kind of show you the integration aspects of data across disparate sensing systems over time, space,
and variables.
If you are interested or it sounds very interesting, please visit our websites.
There are publications and you can ask me questions.
I will probably conclude the slide with the following the presentation with the following picture.
My question for the audience is, do you know where these pictures came from?
These are the pictures of the Hurricane Katrina.
And the reason why I put them up is that they are beautiful, but if you think about it, if you would change the whole scenario and instead of trying to deal with the whole scenario after the hurricane hit the coast, you could have posed the question, could we have stopped the hurricane?
And my gut feeling is that the person, or maybe the next generation like you, would figure out this problem by really building multi-instrument and multi- sensor systems.
And you can imagine how many sensors and instruments you have to deploy to fully understand the whole hurricane so you can figure out how to stop the hurricane as opposed to deal with the consequences of it.
So that would be my concluding remark.
Just something to think about.
Thank you.
If you have questions, please ask them.