HAMK Smart Research Unit open research diary

Edit... Upload asset... Copy asset link template... Copy image template... Copy last modified at... Create another post...

Generative superresolution

2022-05-03, Olli Niemitalo
Last modified: 2022-06-02

🌾 vegetation-sensing 🧮 machine-learning

Open Bioeconomy Week 2022 presentation

Download presentation as PPTX or as PDF.

BIOHILA 2022-06-02 proposal for new HAMK tasks

Download presentation as PPTX.

Context

An approach to vegetation sensing using satellites would be to train a machine learning model to do generative superresolution from satellite images into a resolution that is similar to those obtained by camera drones or even by hand-held cameras. The word generative in this context means “to sample from an approximate distribution of possible realities”. Superresolution means to generate a high spatial resolution image so that at reduced resolution it matches a given low spatial resolution image.

For example, given a rather flat-looking satellite image of a sand-covered desert, extreme superresolution would give us images with individual grains of sand visible, perhaps peppered with sparse vegetation that could be easily detected by machine vision methods. The generated high-resolution images represent an approximate sample of the possible realities that might have resulted in the given low-resolution image. The low-resolution image would be the conditioning input to a conditional generative model. A generative model itself contais a source of randomness that gives rise to variability in the generated sample.

Generative adversarial networks

A generative adversarial network is an artificial neural network implementing a generative model. There is also a variant with least squares loss.

Image channel compatibility woes

Images typically consist of multiple channels. Each channel is associated with a spectral sensitivity function. If two images are from the same camera, then the spectral sensitivity functions match directly. Some imaging satellites and drone cameras also have channels with matching spectral sensitivity functions. After atmospheric correction of the satellite images, the images from the two devices have channel pairs that are compatible in spectral sensitivity. Assuming identical viewing angles (that’s assuming quite a lot!), if we were to reduce the resolution of the drone camera images to those of the satellite image, then those channel pairs should contain images that are identical between the drone and the satellite. This allows to generate synthetic satellite images from the drone images.

However, images from the Sentinel 2 satellite Multi Spectral Instrument have more spectral channels than for example a MicaSense Altum camera. If we were to use only the spectrally matching channels, we’d be throwing away auxiliary information that might improve the realism of the superresolution results. For this reason, the machine learning training data collection should consist of simultaneous acquisition of drone and satellite imagery. This can be done by scheduling the drone imaging missions on satellite flyover days, and by hoping for clear skies.

There are still uses for data from drone-only imaging missins. A separate conditional generative model can be trained on general satellite imagery to convert from a reduced set of channels (just those that match the drone camera channels) to the full set of channels. Given a resolution-reduced drone image as the input, the generated channels can then used as the auxiliary information to train the generative superresolution model, requiring no simultaneous drone and satellite image acquistion.

It is not very clear to me at this stage how this will play out in a probabilistic sense, but hopefully writing it down as math will clarify it.