Imaging Earth from 120,000 feet: Four Years Later.

Just the other day I was wondering how some off the shelf technology had evolved over the past four years. My gold standard is when we flew one of the best and one of the cheapest (less than $500) camera with the largest optical zoom aboard a NASA helium balloon. Four years ago, that camera was the Canon S2 IS (it is the camera in this box). It had a 5MP sensor and a 12X optical zoom and could take a 4GB SD disk. We set out to have a rig that could fire the camera every twenty seconds for the next nine hours (out of twenty, some of these hours are at night!). A little Rube Goldberg'ish set up, but it worked flawlessly:

And we got these beautiful pictures from 120,000 feet. The resolution for this 12X optical zoom at 120,000 feet was pretty like the notch before last in Google Maps as shown in this entry (or about a 1 meter resolution).  Here is a presentation my undergrad students presented then:

GeoCam An offtheshelf Imager for Rapid Response Remote Sensing Monitoring

View and share presentations on PowerShow.com

The site for GEOCAM is here. The attendant blog is here. All the photos can be used for research purposes and featured some interesting elements (like a plane in this shot)

What do we have for the same price four years later ? The Canon SX30IS : a 14.1MP Digital Camera with a  35x wide optical zoom. So about four years later, we can have a camera that is 3 times better in both resolution and zoom. Wow! If we could extrapolate, this camera could provide about 1 foot resolution (30 cm) or probably enough to see a larger sign than this one or this one. In terms of memory, our limit was 8GB (we got a 4GB instead and 1600 photos) but nowadays, you can buy a 32 GB card for less than $100 ( the Transcend 32 GB goes for $65, while the Sandisk 32GB goes for $85) and a 64GB for less than $300 (the SanDisk 64GB goes for $299, the Kingston 64 GB goes for $218 ).4GB got us 1600 photos at 5 MP, so a  32GB SD disk could shoot about 4200 photos and a 64GB could shoot about 8400 photos. Imagine the type of pictures and maps you could get flying this thing at 120,000 feet up. If you are in the U.S. working with some undergrad students you could probably do this as the HASP folks are looking for payloads for the upcoming HASP/NASA balloon. The difference between this balloon and a home made one is that this one will probably fly for 20 hours covering more than 200 miles of terrain. This means that you could probably beat our world record.

From Greg Guzik

Please find attached here the Call for Payloads (CFP) for the September 2011 flight of the High Altitude Student Platform (HASP).  HASP can support up to 12 student payloads (providing power, telemetry and commanding) during a flight to an altitude of 124,000 feet for up to 20 hours.  Details about previous HASP flights and the student payloads flown can be found on the “Flight Information” page of the HASP website at http://laspace.lsu.edu/hasp/Flightinfo-2010.php    Details on the payload constraints and interface with HASP can be found on the “Participant Info” page of the HASP website at http://laspace.lsu.edu/hasp/Participantinfo.php

Applications are due December 17, 2010 and selections will be announced by mid-January 2011.

If you have any questions about the application materials or HASP, feel free to contact us at guzik@phunds.phys.lsu.edu

We will also be conducting a Q&A Teleconference about HASP and the application process on Friday, November 12, 2010 at 10:00 am (central time).  Groups who have previously flown on HASP as well as new organizations should plan on attending this teleconference.  To participate, dial in to 1-866-717-2684 a few minutes prior to the conference time.  When requested enter the conference ID number 6879021 followed by the # key.

Also please forward this e-mail to any others that you fee might be interested in applying.

Cheers,

Greg Guzik

Ps: Please note that we are still intending to fly HASP 2010 payloads during May 2011.

The main reason the photographs taken during our flight are unique revolves around the fact that the data is free for anybody to use and because satellite imagery uses push broom technology as opposed to full square CCDs. The big difference between these types of shots and those taken from low flying drones and planes is the ability to image a larger swath of land (the balloons at an altitude three times higher than the altitude flown by long range commercial aircrafts) and a low cost camera that does not require GPS or compass ( In When You Become the Map I made the case that if you stitch enough photos together then you don't need to be integrated to Google Earth/Maps, you have just produced a new map yourself) making it a potential tool for rapid imaging by the citizenry in case of disaster. We showed that the making of maps could be done very simply by feeding all our stock photos directly to Autopano Pro (that still costs about $100). We actually pushed the envelope of that software (and talked at length with their dev people) then but I think looking at what it can do nowadays, it could handle 8000 x 14 MP photos.

CS: Providing insight on Compressive Sensing, GEOCAM

Sometimes, it is important to take a step back and provide to a wider readership some form of insight about compressive sensing. Let me do that today by connecting the dots between some entries on Nuit Blanche and elements found in the presentation slides of researchers in the field. On the slides for "Testing the Nullspace Property using Semidefinite Programming", Alexandre d'Aspremont, Francis Bach, Laurent El Ghaoui make the excellent point in slide 4 that:

• Sparsity is a proxy for power laws. Most results stated here on sparse vectors apply to vectors with a power law decay in coefficient magnitude.
• Power laws appear everywhere. . .

and this is indeed what I have tried to elaborate in the Sparsity in Everything series of entries. One the same subject, a new idea is also emerging that says that power-laws do not fit well with outliers particularly on the high end of it. Didier Sornette, in his recent arxiv preprint entitled Dragon-Kings, Black Swans and the Prediction of Crises thinks there is a positive feedback mechanism that produces even larger elements making them much sparser. Could compressive sensing be used to detect events with positive feedbacks out of the many unpredictable ones that fit a power law and therefore are much smaller in effects ?

Terry Tao also has a new presentation on Compressive Sensing where one can find this nugget:

An analogy would be with the classic twelve coins puzzle: given twelve coins, one of them counterfeit (and thus heavier or lighter than the others), one can determine the counterfeit coin in just three weighings, by weighing the coins in suitably chosen batches. The key point is that the counterfeit data is sparse.

It also looks like this example helps journalists and the public at large get the idea on group testing as the example is used in this article on Terry's series of lectures in Australia. You may recall a similar example and treatment here on this blog of that problem with balls instead of coins.

Further in the presentation, he also makes the more cryptic statement :

There are now several theoretical results ensuring that basis pursuit works whenever the measurement matrix A is sufficiently “incoherent”, which roughly means that its matrix entries are uniform in magnitude. (It’s somewhat analogous to how the secret to solving the twelve coins problem is to weigh several of the coins at once.)

Finally, on a totally different note, here is a presentation done by some NASA contractor folks that uses the same name as our 2005 GEOCAM project and uses the same concepts. Low tech cameras and an aerial capability can provide real time data in case of catastrophes. My students and I worked on this as a student project a little bit after what happened to New Orleans with Katrina. All the photos taken during that flight are here and were assembled using a low cost off-the-shelf software called Autopano Pro ( that uses SIFT markers) to produce these beautiful maps. Eventually, we never got any interest by other parties about what we had done. I am very glad the idea is continuing to live on.

Imaging from the sky: When You Become The Map

There is a new call from the HASP folks about submitting new payloads to be flown next year in a NASA high altitude balloon. Deadline is December 18, 2007, it is directed toward undergraduate projects.
From the HASP website:

September 17, 2007: HASP CALL FOR PAYLOADS 2007-2008 RELEASED: The HASP Call for Payloads 2007-2008 (CFP) has been released and application materials are now available on the HASP website “Participant Info” page. Student groups interested in applying for a seat on the September 2008 flight of HASP should download these materials and prepare an application. New for this year is an increase in the allowed weight of the student payloads. Small class payloads can now mass up to 3 kilograms and large class payloads can weigh as heavy as 20 kilograms. Applications are due December 18, 2007 and selections will be announced by mid-January 2008.

The photos below and sideways are a 10 percent composite of several photos taken at 30,000 feet, with a 3x optical zoom at 500 mph. The speed makes it very unlikely to get any good details without some type of processing. And so, for the time being, imaging the ground with some type of precision with some type of point and shoot camera seems to be only feasible to payloads on balloons.
Compared to satellite imagery, one of the interesting capability is to remove the effect of clouds when possible. In satellite imagery, cameras work with pushbroom technology where the imager is a line of pixels (not a square dye). One consequence is the inability of photographing twice the same object with one sweep. Using off the shelf cameras on much slower balloons allow one to obtain multiple images of the same object at different angle. This is important when one wants to evaluate whether the object is noise or not.

Chris Anderson of the Long Tail book mentioned a different approach by Pict'Earth to using images from the sky using UAVs and patching them into Google Earth. This is interesting, but as I have mentioned before, when you take enough images, you don't need Google Earth, you don't need the headache of re-projecting these images onto some maps (even though it looks easier with Yahoo Map Mixer for small images), because you are the map. No need for IMUs or GPS instrumentation. This is clearly an instance of advances in stitching algorithms removing hardware requirements on the sensors. As for the current results Chris is getting from PTGui, I am pretty sure the autopano folks will enable the orthographic projection soon in order to cater to that market. With balloons, the view is from very far, so the patching algorithm has no problem stitching images together. In the case of UAVs, you need the orthographic projections.

Eventually, two other issues become tremendously important (especially in the context of Search And Rescue). Cameras and memory are going cheaper and one is faced with GB's of data to store, map and share. Our experience is that the sharing is challenging when you go over 2 GB of data mostly because of small file format limits (2 GB). Zoomify is interesting and they need to figure out a way to deal with larger images. While Autopano allows for images taken at different times to be overlayed with each other (a very nice feature), the viewer might be interested in this time information. Right now I know of no tool that allows one to switch back and forth between different times for the same map.

References:
1. Comparing Satellite Imagery and GeoCam Data
2. A 150-km panoramic image of New Mexico

Search and Rescue: New Directions.

After the heartbreaking results of GeoCam and Hyper-GeoCam during HASP 2007, we are going to investigate the same type of technique from less unusual ways of putting things in the air. Some of our initial findings can be seen here. We cannot afford to wait for a year to have results like these (small planes,...). In particular, the inability to get systems in a working shape and actually taking data in a rapid turn around is just a big invitation to Murphy's law.
I have already tried to gather similar data from commercial airliners but with a 3X optical zoom point and shoot cameras. I am going to improve that. Since we are talking about an altitude of 10,000 feet and a speed of about 700 km/h, the parameter for map/panorama making are different. The image in this entry was taken over Canada and assembled about 10 photos. In this example there was no attention given to the detail of the scene.

It looks there is some interest from other people in this area (I had no idea), I am going to investigate that as well as with some of the contact I had during the search for the Tenacious. I'll report on this later. One of the most surprising findings of the current search for Steve Fossett is the finding of the location of at least six other crashes. I had known that crashes occured and went missing after a year but I personally had no idea of the the large amount of missing planes:

The search has spread across an area of 17,000 square miles, twice the size of New Jersey. Crews will continue combing sections of that vast landscape, but on Sunday they began focusing on the territory within 50 miles of the ranch. Most crashes occur within that radius during takeoffs or landings, Nevada Civil Air Patrol Maj. Cynthia Ryan said.

``We've got close to 100 percent covered, at least in some cursory fashion,'' Ryan told reporters Sunday. ``We have to eliminate a lot of territory.''

The discovery of at least six previously unknown wrecks in such a short time has been a stark demonstration of the odds against finding Fossett's single-engine Bellanca Citabria Super Decathlon.

The Florida-based Air Force Rescue Coordination Center, which is helping coordinate the search, maintains a registry of known plane wreck sites.

The registry has 129 entries for Nevada. But over the last 50 years, aviation officials estimate, more than 150 small planes have disappeared in Nevada, a state with more than 300 mountain ranges carved with steep ravines, covered with sagebrush and pinon pine trees and with peaks rising to 11,000 feet.

What is currently also very clear in my mind is that the turn-around between instrument data gathering and analysis is taking too long. The Mechanical turk initiative is a noteworthy one, however, it does not address our current inability to process intelligently the wall of data coming from these hyperspectral imagers (which seem to largely never recover any useful data for the searches). I am thinking of probably using some of the compressed techniques to be able to do that on-board the planes. Having to these data used to be difficult, it looks like the European Space Agency understands that more people need to have access to them to find interesting things in niche markets. They make their data available here.

Since the number of posts on the subject has risen over the course of this year, I am summarizing all these entries in a more coherent way here. You can also reach that link by clicking on the right side bar. In that link, there are several subjects that do not have an entry but eventually I want to address them and the necessary improvements needed for the technology to be optimal in terms of operations.

This is how science is done, trial and error in the mud.

As noted by the HASP folks, one of our payload fell into the mud after landing from a 37 km fall with a parachute. We just got the cameras this morning and found out it was GeoCam. Hyper-GeoCam is fine. Now we need to open the boxes and find out if there is anything in either of them. This is Science in Motion where Murphy's law always strike. The camera look fine, let's see if they actually took pictures.
[Update: GeoCam is OUT.]

Ready to launch

the title says it all. HyperGeoCam will be lifting off in a short while over New Mexico for about 10 to 20 hours taking about 1000 snapshots if all goes well.

[ Update: it went up at 7:30 am mountain time, 8:30 Central time, 15:30 UMT, you can see LIVE what one of the webcam see here]

[ Update 2: HASP has landed. 6.55 miles from the California border after 19 hours of flight. woohoo. We won't know if we have data until the hardware is returned to us.]

Implementing Compressed Sensing in Applied Projects

We are contemplating using Compressed Sensing in three different projects:

• The Hyper-GeoCam project: This is a payload that will be flown on the HASP platform in September. Last year, we flew a simple camera that eventually produced a 105 km panorama of New Mexico. We reapplied for the same program and have been given the OK for two payloads. The same GeoCam will be re-flown so that we can produce a breath taking panorama from 36 km altitude. The second payload is essentially supposed to be a hyperspectral imager on the cheap: i.e. a camera and some diffraction gratings allowing a fine decomposition of the reflected sun light from the ground. The project is called Hyper-GeoCam and I expect to implement a random lens imager such as the one produced at MIT. Tests will be performed on the SOLAR platform.
  
• The DARPA Urban Challenge: We have a car selected in the track B: We do not have Lidars and need to find ways to navigate in an urban settings with little GPS availability. The autonomous car is supposed to be navigating in a mock town and follow the rules of the California traffic laws, that includes passing other cars.
  
• Solving the Linear Boltzmann equation using compressed sensing techniques:The idea is that this equation has a known suite of eigenfunctions (called Case eigenfunctions) and because they are very difficult to use and expand from, it might be worth a try to look into the compressed sensing approach to see if it solves the problem more efficiently.
  

Compressed Sensing Hardware Implementations

[Update Nov. '08: Please find the Compressed Sensing Hardware page here ]

The one pixel camera made by Richard Baraniuk and his group at Rice is the one that received the most press. In order to build this new type of camera ones needs a DMD controlled board (at 6K$), one pixel and you're set. That one pixel detector could really be a photodiode, a radiation detector or other Teraherz receiver.

However, the concept of compressed sensing does not really need hardware transformation/implementation as impressive as this one. At one end, of the spectrum, one can just change the way sampling is done as in MRI work ( Sparse MRI: The application of compressed sensing for rapid MR imaging by Michael Lustig, David Donoho, and John M. Pauly), where sampling in the k-space is done below the Nyquist criterion. Current technology already allows for the gathering of signals with less than pure frequency content. Using the ability to put several frequencies together in one spike, allows one to retrieve directly compressed samples thereby enabling substantial savings in acquisition time.

In between the ends of that spectrum are several set-ups that are trying to use current hardware with some slight modification to provide lower sampling sets. I have come across two: Random Lens Imagers and Compressive Sampling Spectrometers.


* The random lens imaging technique developed at MIT by Rob Fergus, Antonio Torralba, and William T. Freeman is where one uses a normal DSLR camera but removes the lens and replaces it with a transparent material in which mirrors where included at random. The point they are making is the following: The image obtained from this system are the compressed sensing measurements. In order to reconstruct the original image, they have to calibrate the new camera set-up. The only way to do this is by having say a laser light shone on the camera and see how the ray is being imaged. When you have a 10 Mp camera, this means that you will shine that laser light from ten millions locations in the field of view in order to have the ten million unit response to each laser light. When that is done, you solve a linear algebra problem (a simple matrix inversion) to obtain the calibration/response matrix. Then every time, you take a picture, you multiply your result with that matrix and obtain the picture you were looking for. It is pretty obvious that the calibration step could be improved by removing the need to shine a laser light ten million times. In other words, you have too many unknowns for too few equations (being lazy you will not shine that laser light ten millions times). The compressed sensing theory really says that you can solve that problem having too few equations (or calibration image ) so that you can find the inverted matrix with much fewer trials. The advantage is the potential ability to provide more information from current CMOS/CCDs. In effect, by using many calibration images, one could potentially obtain superresolution information (smaller than the pixel size of the DSLR) or depth information. The ability to use the current very large CMOS capability (instead of one pixel) has a very real potential. Consider this: if a 1 MP camera can produce 30 KB images in JPEG, there is 30 000 information allow a good representation of a 2-D scene (about 170 information per dimension.) Therefore, a 3-D scene would require about 4.9 MB for a good description. Clearly one is already gathering that much information from normal cameras.

* Compressive sampling imager: With most hyperspectral cameras there is the need to reduce the amount of information gathered while it is being gathered not after. For instance, the Hyperion camera on EO-1 was designed with the bandwidth for transmitting the information down in mind (the TDRSS system cannot handle more than 6 MBit/s) that is why people try to compress the data after it has been gathered in order to allow efficient transmission of signals. But on a spacecraft like EO-1 you don't really have that much computational power and you are really looking for a way to acquire the minimum amount of information in the first place. The current interesting undertaking in this field seems to be about compressed sensing spectrometers or compressive sampling spectrometers by the DISP group at Duke (David J. Brady, Mike Gehm, Scott McCain, Zhaochun Xu, Prasant Potuluri, Mike Sullivan, Nikos Pitsianis, Ben Hamza, Ali Adibi).


A good presentation of their effort can be found here. The idea is that if you assume that:
• Measurements are expensive
• Photons are scarce and
• Spectra are sparse
you can modify your current set-up by introducing a mask between two gratings. So instead of having systems that respond to single spectral signals, group testing is used so that several spectral bands can provide one compressed measurement.



More advances on this type of hardware can be found in this fascinating article where it shows why Nyquist and Golay sampling theorems were reduced to piece by current advances, this a must read.

It is pretty obvious that any of these elements for adopting compressed sensing fit pretty well with radiation measurements where one can direct radiation beams. I am also thinking of implementing some of these systems for the HASP 2007 flight.

Sharp panoramic view of the tiny

It looks like making panoramas is not just for the big stuff. I just came accross an interesting paper on image stitching and microscopy. They do not seem to have evaluated algorithms based on SIFT and Ransac to do the stitching like Autostich or Autopano Pro. In the paper, Thevenaz and Unser mention that they obtain some type of superresolution. In this field, one is interested in removing unfocused objects whereas in other fields, some people are interested in surprising occurences (like a jet flying by) or in inserting new artifacts.

HASP 2007

We now have two payloads flying on HASP this coming fall: A re-flight of GeoCam and the new Hyper-GeoCam.

Reflecting on a moving view

I recently flew on a small shuttle plane between College Station and Houston and wanted to see what the view looked like in a panorama. Stitching several photos together using Autopano Pro, this is what I eventually saw from 4,000 feet. This is a little bit different than say 112,000 feet since the angle to the scence of interest makes it difficult to patch images together. When the camera is close to the land, the movement of the plane changes quite rapidily the angle to the scene and it then becomes difficult for the software to patch images together.

We are slowly getting there

For the past two months, we have been talking to one of the maker of Autopano Pro so that we could process a very large panorama of about 609 images (each of which is 3.2 Mpixels large). The current beta release (version 1.3 RC 3) still has a memory leakage problem for very large panoramas like this one. This monster is about 2 GB large. One of the smaller panorama can be found here. Others can be found here.

Space Vantage Points

When Nadar went up his balloon 150 years ago and took the first aerial photo, he triggered Jules Vernes into writing "Five weeks in a balloon".
The first panoramic view of a natural disaster was taken one hundred years ago after the 1906 San Francisco Earthquake using a kite.

The first photo taken from space was obtained in 1946, from a V2 rocket and the first panorama in space was taken two years later.

Fifty years later, thanks to the HASP platform, we took one of the longest panoramic view from a high altitude balloon. This is mainly due to our ability to store large amount of data in common cameras (4GB).
. The interesting aspect of our approach relies on the fact that our low cost camera does not need to be equipped with either a GPS or an Inertial Navigational Unit: The use of a software like Autopano Pro enables us to patch automatically all of our pictures together and create a single large map.

The slashdot effect

This Slashdot announcement for the preliminary results on GeoCam generated about 25,000 viewers in the course of two days. While the comment section on Slashdot is not that useful, we received some very helpful tips and ideas on our blog. I have stitched together some of these panaromas with one of the non-free version of the stitching algorithm and I am very impressed. The current panoramas are 25 to 30 MB large.

Panoramic views from 36 kms high

I just used Matthew Brown's AutoStitch program to generate panoramas from our GeoCam flight.

Cognitive Convergence

I am not blogging that much these days for several reasons. First, I am looking into how we can use some of the artificial intelligence techniques we are developing for the DARPA Urban Grand Challenge to diagnose and explore autism. I recently attended a talk by Hideki Kozima who uses of small robots like Keepon to evaluate the socialization of autistic kids. Take a look at the video here. After five to ten sessions with the Keepon, he could show the beginning of a joint attention development in autistic kids. This was very impressive.

I am also involved in the processing of the data we just received from our GeoCam on HASP. We have 4 GB of data to share with the rest of the world. We are developing a strategy on how to do this efficiently.

We are considering a run for the DARPA Urban Grand Challenge with Pegasus Bridge 1. Only this time, the mechanical aspect of the project will take a back seat to other technologies we are developing. We are interested in driver's gaze recording and supervised learning of road driving behavior. But more on this later...

GeoCam: An Off-The-Shelf Imager for High Altitude Balloons

The blog for the GeoCam Imager is up and can be found here. Here is the most recent photo of it in its casing.



The whole reason as to why we think this off-the-shelf camera is a good idea is explained here.

Fabricating elements around current high end consumer digital cameras (like the clicker) is the most effective way of utilizing advances in digital camera development. We intend on showing how we did all this on sites like Hackaday. Thank you to the HASP program at LSU for giving us a seat in the High Altitude Student Platform (HASP).

Going high

Our proposal for the Geocam has been selected to fly on-board a high altitude balloon campaign organized by the Louisiana Space Consortium. It will be launched from Fort Sumner, NM. Movies taken from high altitude balloons can be found at the University of Montana within the BOREALIS High Altitude Balloon program.

We expect to use an JVC Everio Hard Disk Camcorder in order to fit our tiny requirements.

New Orleans Google Map Before and After Katrina

Google Maps has satellite imagery of New Orleans taken on September 1, 2005. They can be found here.