Core vision - TensorImageMS

Basic functionality to represent and visualize multi-spectral data images

---

source

TensorImageMS

 TensorImageMS (x, **kwargs)

Class to represent multi-spectral data (more than 3 channels)

The class is created with the tensor of ordered channels as input. It requires parameters bands and brgtX (which will be introduced shortly) and therefore should only be constructed with the following factory methods

---

source

TensorImageMS.from_tensor_bands

---

source

TensorImageMS.from_tensor

@patch(cls_method=True)
def _for_test(cls: TensorImageMS, x):
    return cls.from_tensor(x, bands=[], captions=[], brgtX=[])

t = TensorImageMS._for_test([[1.,2.],[.3,.4],[5.,6.]])

test_eq(t, TensorImageMS([[1.,2.],[.3,.4],[5.,6.]]))
test_eq(t.bands,[])
test_eq(t.brgtX,[])

Multiple images - one for each set of channels

Our approach is to display multiple images for each multi-spectral image. We can either present each channel as a single “monochrome” image, or more compactly, sets of 3 channels as one “false color” image (or as a true color image when we display the actual RGB channels).

This is done by providing a tuple of indices corresponding to the required channels. In practice, the tuple will either be of length 3 (false color) or 1 (monochrome). A list of these “channel tuples” is in the bands attribute.

The number of images in the visualization can be calculated from the list.

---

source

TensorImageMS.num_images

 TensorImageMS.num_images ()

test_eq(t.num_images(),0)

Each individual image in the visualization is represented by a “filtered” tensor that represents only the selected band(s). The following method does the selection.

t3 = t._select_bands((1,0,2))

test_eq(t3,TensorImageMS([[.3, .4],[1., 2.],[5.,6.]]))
test_eq(t3.bands,t.bands)
test_eq(t3.brgtX,t.brgtX)

t1 = t._select_bands((1,))
test_eq(t1,TensorImageMS([[.3, .4]]))

Image Brightening

Another practical problem that needs to be addressed when dealing with normalized Sentinel 2 images is that typical pixel values are very “dark” when rendered graphically.

It is helpful to artificially brighten normalized tensors, using multipliers (that can vary according to the band). These multipliers lists are referenced by the brgtX attribute of our class.

Test calculation

t1b2 = t1._brighten([2.])
test_eq(t1b2, TensorImageMS([[0.6, 0.8]]))

Use a multiplier of 1.0 when brightening is not called for.

t1b1 = t1._brighten([1.])
test_eq(t1b1, t1)

Display on grid

The brightened images are then displayed in the contexts that represent a matplotlib grid of images.

The grid consists of rows, one for each MS image. Each row consists of all the individual images (one per column) corresponding to channel sets.

This is all put together to display the final image(s).

---

source

TensorImageMS.show

 TensorImageMS.show (ctxs=None, **kwargs)

Example

We use the MS file io functionality defined here to load a multi spectral image tensor.

def get_input(stem: str) -> str:
    "Get full input path for stem"
    return "./images/" + stem

def tile_img_name(chn_id: str, tile_num: int) -> str:
    "File name from channel id and tile number"
    return f"Sentinel20m-{chn_id}-20200215-{tile_num:03d}.png"

def get_channel_filenames(chn_ids, tile_idx):
    "Get list of all channel filenames for one tile idx"
    return [get_input(tile_img_name(x, tile_idx)) for x in chn_ids]

from fastgs.test.io import *

def load_tensor(tile_num: int):
    bands=[(2,1,0),(3,),(1,),(0,)]
    captions=["B04,B03,B02","B8A","B03","B02"]
    brgtX=[[3.75,4.25,4.75],[2.5],[4.25],[4.75]]
    files=get_channel_filenames(["B02","B03","B04","B8A"],tile_num)
    t=read_multichan_files(files)
    return TensorImageMS.from_tensor(t,bands=bands,captions=captions,brgtX=brgtX)

The tensor is loading channels with ids “B02” (Blue - B), “B03” (Green - G), “B04” (Red - R) and “B8A” (Near Infra Red - NIR) into channel indices 0, 1, 2 and 3.

The bands represent channels sets with indices (2,1,0) i.e. (R, G, B) and (3) i.e. (NIR) and (1) i.e. (G) and (2) i.e. (B). The first set is shown as an RGB image with R,G and B actually corresponding to the “real” Red, Green, and Blue channels. If other channels had been selected, it would be a “false colour” image.

The brgtX represents a brightness multiplier to multiply the values in each channel so that the result is not visually dark.

When the image is displayed, it actually produces a sequence of 4 images, a colour image corresponding to RGB, and 3 monochrome images, each corresponding to NIR, G and B respectively.

msimg=load_tensor(66)
msimg.show()

[<AxesSubplot:title={'center':'B04,B03,B02'}>,
 <AxesSubplot:title={'center':'B8A'}>,
 <AxesSubplot:title={'center':'B03'}>,
 <AxesSubplot:title={'center':'B02'}>]

Animating multiple images

Another possibility for display of the individual images of channel tuples is to create an animation that cycles through the images, rather than laying them out as a row of images.

Note that this animation has a runtime dependency on ffmpeg which is not listed in the package dependency. You will need to install it manually for the animation to function.

msimg._show_animation()

This reduces the amount of visual space taken up by the image, and also makes it easier to see how an individual pixel/area changes “color” as the channel set changes.

However, since this creates an embedded HTML movie, it results in a large increase in the cell output size. It might be interesting if we could produce in-place “.gif” files instead.