Not logged in!

Image

The Image class allows you to work with image data
Introduction

The Image class deals with handling image data, allowing you to display radio/dropdown/checkbox lists or textboxes alongside an image.

Contents

Image.View

Image.View allows you to output a list of images, optionally with text for each image.

Example Applications:

  • Displaying the outputs of an image captioning system

Example Usage
1import feather as ftr
2from my_model import my_image_captioning_model
3
4def init():
5    return ftr.File.Upload(types=["images"], title="Upload images for image captioning")
6
7def run_image_captioning(uploader):
8    images = uploader.get_image_files() # equivalent to 'uploader.get(format="images")'
9    # images = [
10    #     {"name": "image1.jpg", "data": np.ndarray},
11    #     {"name": "image2.png", "data": np.ndarray}
12    # ] 
13      
14    captions = my_image_captioning_model(images) # ["A cat sitting in a sink", "A dog catching a frisbee"]
15
16    #################### CONSTRUCTOR ####################
17    return ftr.Image.View(images=images, output_text=captions)
18    #####################################################
19
20if __name__ == "__main__":
21    bundle = ftr.bundle(code_files=[__file__, "my_model.py"], model_files=["outputs/model.ckpt"])
22    ftr.build(name="Image Captioning Model", init=init, steps=[run_image_captioning], file_bundle=bundle)

Constructor:
  • images: ImageInputType - a list of images (see ImageInputType) to be shown to the end user
  • output_text: Optional[ListType[str]] = None - an optional list of output texts to be shown for each image. If provided, length of output_text must be the same as the length of images
  • title: Optional[str] = None - a title semantically attached to the component
  • description: Optional[str] = None - a description semantically attached to the component

Attributes:
  • images - get the images passed into this component

Component Playground:
Arg NameValue
images
output_text
title
description
ftr.Image.View(
    images=[
    {
        "name": "cat.jpg",
        "data": "<VALID NUMPY IMAGE ARRAY (HxWx3)>"
    },
    {
        "name": "dog.png",
        "data": "<VALID NUMPY IMAGE ARRAY (HxWx3)>"
    }
], 
    output_text=["A cat sitting in a sink", "A dog in a field with flowers"], 
    title="Image Captioning outputs", 
    description="Click an image on the left to view its caption")
Image.WithSelectMulti

Image.WithSelectMulti allows a user to select multiple items for one image, over a list of images

Example Applications:

  • An image captioning model which conditions its captions on multiple objects
  • Selecting one or more transformations to apply to an image

Example Usage
1import feather as ftr
2from my_model import my_object_detector, my_conditional_caption_model
3
4def init():
5    return ftr.File.Upload(types=["images"], title="Upload images for image captioning")
6    
7def run_object_detection(uploader):
8    images = uploader.get_image_files() # equivalent to 'uploader.get(format="images")'
9    # images = [
10    #     {"name": "image1.jpg", "data": np.ndarray},
11    #     {"name": "image2.png", "data": np.ndarray}
12    # ] 
13
14    detected_objects = my_object_detector(images) 
15    # detected_objects = [["cat", "sink", "gloves", "sponge"], ["dog", "frisbee", "person"]]
16    
17    #################### CONSTRUCTOR ####################
18    image_checkbox = ftr.Image.WithSelectMulti(images, lists=detected_objects,
19                                                title="Select which objects to condition a caption on", 
20                                                description=None)
21    #####################################################
22
23    return image_checkbox
24      
25def run_captioning(image_checkbox):
26    #################### ACCESSOR ####################
27    selected_objects = image_checkbox.get_selected() # equivalent to 'image_checkbox.get()'
28    ##################################################
29    # selected_objects = [["cat", "sink"], ["person"]]
30
31    images = image_checkbox.images
32    captions = my_conditional_caption_model(images, selected_objects)
33    # captions = ["A black and white cat sitting in a sink", "A long haired lady in a field playing with her dog"]
34
35    return ftr.Image.View(images, output_text=captions)
36
37if __name__ == "__main__":
38    bundle = ftr.bundle(code_files=[__file__, "my_model.py"], model_files=["outputs/object_detection_model.ckpt", "outputs/image_captioning_model.ckpt"])
39    ftr.build(name="Conditional Image Captioning Model", init=init, steps=[run_object_detection, run_captioning], file_bundle=bundle)
Example Usage
1import feather as ftr
2from my_model import my_style_transfer_model
3            
4def init():
5    return ftr.File.Upload(types=["images"], title="Upload images for style transfer")
6
7def collect_options(uploader):
8    images = uploader.get_image_files() # equivalent to 'uploader.get(format="images")'
9    # images = [
10    #     {"name": "image1.jpg", "data": np.ndarray},
11    #     {"name": "image2.png", "data": np.ndarray}
12    # ] 
13
14    ##### We can optionally provide a list of values as selected by default
15    default_selected = [("Cartoonise", True), ("Gender swap", False), ("Picasso style", False)] * len(images)
16    #####
17    
18    #################### CONSTRUCTOR ####################
19    image_checkbox = ftr.Image.WithSelectMulti(images, lists=default_selected,
20                                                title="Choose which styles to apply to your images", 
21                                                description=None)
22    #####################################################
23
24    return image_checkbox
25      
26def run_style_transfer(image_checkbox):
27    #################### ACCESSOR ####################
28    selected_objects = image_checkbox.get_selected()
29    ##################################################
30    # selected_objects = [["Cartoonise"], ["Cartoonise", "Picasso style"]]
31
32    images = image_checkbox.images
33    output_images = my_style_transfer_model(images, selected_objects)
34    # output_images = [np.ndarray, np.ndarray]
35
36    return ftr.Image.View(output_images)
37    
38if __name__ == "__main__":
39    bundle = ftr.bundle(code_files=[__file__, "my_model.py"], model_files=["outputs/model.ckpt"])
40    ftr.build(name="Style Transfer Model", init=init, steps=[collect_options, run_style_transfer], file_bundle=bundle)

Constructor:
  • images: ImageInputType - a list of images (see ImageInputType) to be shown to the end user
  • lists: Union[ListType[ListType[str]], ListType[ListType[Tuple[str, bool]]]] - a list of lists (or tuples) containing the items for a end user to select from, for each image. The length of lists must be the same as the length of images. This parameter can either be a list of lists, where checkbox values default to false, or a list of tuples of (<item>, <boolean>) where you can specify which items are selected by default.
  • title: Optional[str] = None - a title semantically attached to the component
  • description: Optional[str] = None - a description semantically attached to the component

Attributes:
  • images - get the images passed into this component

Accessors:
  • get_selected() -> ListType[ListType[str]] - a list of lists which contain the items which were selected by the end user
  • get_all() -> ListType[ListType[Tuple[str, bool]]] - a list of tuples for each image, in format (<item>, <boolean>), which contains the item value, and a boolean to indicate whether that item was selected or not.
  • get(return_all=False) - see below
get() (usage of .get() is not recommended)
def get(return_all=False):
    if return_all:
        return get_all()
    return get_selected()

Component Playground:
Arg NameValue
images
lists
title
description
ftr.Image.WithSelectMulti(
    images=[
    {
        "name": "cat.jpg",
        "data": "<VALID NUMPY IMAGE ARRAY (HxWx3)>"
    },
    {
        "name": "dog.png",
        "data": "<VALID NUMPY IMAGE ARRAY (HxWx3)>"
    }
], 
    lists=[[("cat", True), ("sponge", False), ("sink", True)], [("dog", False), ("grass", True), ("tree", True)]], 
    title="Select objects", 
    description="Select objects you want a caption generated/conditioned on")
Image.WithSelectOne

Image.WithSelectOne allows a user to select one item per image (from a list of items), over a list of images

Example Applications:

  • An image captioning model which conditions its captions on a single object

Example Usage
1import feather as ftr
2from my_model import my_object_detector, my_conditional_caption_model
3            
4def init():
5    return ftr.File.Upload(types=["images"], title="Upload images for image captioning")
6
7def run_object_detection(uploader):
8    images = uploader.get_image_files() # equivalent to 'uploader.get(format="images")'
9    # images = [
10    #     {"name": "image1.jpg", "data": np.ndarray},
11    #     {"name": "image2.png", "data": np.ndarray}
12    # ] 
13
14    detected_objects = my_object_detector(images) 
15    # detected_objects = [["cat", "sink", "gloves", "sponge"], ["dog", "frisbee", "person"]]
16    
17    #################### CONSTRUCTOR ####################
18    image_selector = ftr.Image.WithSelectOne(images, lists=detected_objects, style="radio",
19                                            title="Select which objects to condition a caption on", 
20                                            description=None)
21    #####################################################
22
23    return image_selector
24      
25def run_captioning(image_selector):
26    #################### ACCESSOR ####################
27    selected_object = image_selector.get_selected() # equivalent to 'image_selector.get()'
28    ##################################################
29    # selected_object = [["sink"], ["frisbee"]]
30
31    images = image_select.images
32    captions = my_conditional_caption_model(images, selected_object)
33    # captions = ["A sink with a cat and dishes in it", "A frisbee about to be caught by a dog"]
34
35    return ftr.Image.View(images, output_text=captions)
36    
37if __name__ == "__main__":
38    bundle = ftr.bundle(code_files=[__file__, "my_model.py"], model_files=["outputs/object_detection_model.ckpt", "outputs/image_captioning_model.ckpt"])
39    ftr.build(name="Conditional Image Captioning Model", init=init, steps=[run_object_detection, run_captioning], file_bundle=bundle)

Constructor:
  • images: ImageInputType - a list of images (see ImageInputType) to be shown to the end user
  • lists: ListType[ListType[str]] - a list of lists containing the items for a end user to select from, for each image. The length of lists must be the same as the length of images.
  • style: Literal["radio", "dropdown"] = "radio" - whether to display the items as radio buttons or dropdowns. Defaults to radio buttons
  • title: Optional[str] = None - a title semantically attached to the component
  • description: Optional[str] = None - a description semantically attached to the component

Attributes:
  • images - get the images passed into this component

Accessors:
  • get_selected() -> ListType[str] - a list which contains the item for each image selected by the end user
  • get_selected_with_index() -> ListType[Tuple[str, int]] - a list of tuples, in format (<item>, <index>). Index indicates which item index the string value was in the list provided to the constructor
  • get(return_indices=False) - see below
get() (usage of .get() is not recommended)
def get(return_indices=False):
    if return_indices:
        return get_selected_with_index()
    return get_selected()

Component Playground:
Arg NameValue
images
lists
style
title
description
ftr.Image.WithSelectOne(
    images=[
    {
        "name": "cat.jpg",
        "data": "<VALID NUMPY IMAGE ARRAY (HxWx3)>"
    },
    {
        "name": "dog.png",
        "data": "<VALID NUMPY IMAGE ARRAY (HxWx3)>"
    }
], 
    lists=[["cat", "sponge", "sink"], ["dog", "grass", "tree"]], 
    style="radio",
    title="Select an object", 
    description="Select one object you want a caption generated/conditioned on")
Image.WithTextIn

Image.WithTextIn allows a user to enter some text for each image

Example Applications:

  • Visual Question Answering

Example Usage
1import feather as ftr
2from my_model import my_vqa_model
3            
4def init():
5    return ftr.File.Upload(types=["images"], title="Upload images for image captioning")
6
7def collect_questions(uploader):
8    images = uploader.get_image_files() # equivalent to 'uploader.get(format="images")'
9    # images = [
10    #     {"name": "image1.jpg", "data": np.ndarray},
11    #     {"name": "image2.png", "data": np.ndarray}
12    # ] 
13
14    default_text = ["What colour is the main object?"] * len(images)
15    #################### CONSTRUCTOR ####################
16    question_inputs = ftr.Image.WithTextIn(images, text=default_text, max_chars=256,
17                                        title="Enter a question for each image!", 
18                                        description=None)
19    #####################################################
20
21    return question_inputs
22      
23def run_vqa(question_inputs):
24    #################### ACCESSOR ####################
25    questions = question_inputs.get_text() # equivalent to 'question_inputs.get()'
26    ##################################################
27    # questions = [["How many horses are in the picture?"], ["What does the sign in the background say?"]]
28
29    images = question_inputs.images
30    answers = my_vqa_model(images, questions)
31    # captions = ["4", "feather is awesome"]
32
33    return ftr.Image.View(images, output_text=answers)
34
35if __name__ == "__main__":
36    bundle = ftr.bundle(code_files=[__file__, "my_model.py"], model_files=["outputs/model.ckpt"])
37    ftr.build(name="Visual Question Answering Model", init=init, steps=[collect_questions, run_vqa], file_bundle=bundle)

Constructor:
  • images: ImageInputType - a list of images (see ImageInputType) to be shown to the end user
  • default_text: Optional[Union[str, ListType[str]]] = None - optional default text to be displayed in the text input boxes. If entered, can be of type str or a list of str. If str is provided, the string is duplicated for all the images. If a list of str is provided, the length text must equal the length of images
  • max_chars: Optional[int] = 256 - an optional argument which specifies the maximum amount of characters a user can enter in a text box.
  • title: Optional[str] = None - a title semantically attached to the component
  • description: Optional[str] = None - a description semantically attached to the component

Attributes:
  • images - get the images passed into this component

Accessors:
  • get_text() -> ListType[str] - a list which contains the texts an end user has entered.
  • get() - see below
get() (usage of .get() is not recommended)
def get():
    return get_text()

Component Playground:
Arg NameValue
images
default_text
max_chars
title
description
ftr.Image.WithTextIn(
    images=[
    {
        "name": "cat.jpg",
        "data": "<VALID NUMPY IMAGE ARRAY (HxWx3)>"
    },
    {
        "name": "dog.png",
        "data": "<VALID NUMPY IMAGE ARRAY (HxWx3)>"
    }
], 
    default_text=["A <MASK> sitting in a sink", "A dog <MASK> in a field"], 
    max_chars=256,
    title="Enter <MASK> tokens", 
    description="Enter masked tokens for multimodal language modelling")
Image specific types
# N.B. "ListType" in our documentation refers to the standard Python List type
  # (so as not to be confused with our 'List' class)

class ImageInputObj():
    name: str
    data: np.ndarray

ImageInputType = Union[ListType[ImageInputObj], ListType[np.ndarray]]