Armory provides the functionality to save benign and adversarial examples to the Armory output directory. The type of output file you receive depends on the scenario, as indicated in the table below:
| Scenario | Output File Type(s) | |
|---|---|---|
| audio_asr | .wav | |
| audio_classification | .wav | |
| carla_video_tracking | .png, .mp4 | |
| dapricot_scenario | .png | |
| image_classification | .png | |
| object_detection | .png, .json | |
| video_ucf101_scenario | .png, .mp4 |
Armory also exports a pickle file containing ground-truth labels, benign predictions, and adversarial predictions.
To enable data export, in the "scenario" section of the configuration, set the "export_batches" field to True. This will result in exporting all batches. If you do not want to export data, you can set this field to False or simply omit it from the configuration; see configuration_files.md.
Here is an example where data will be exported:
"scenario": {
"kwargs": {},
"module": "armory.scenarios.image_classification",
"name": "ImageClassificationTask",
"export_batches": True
},
If you are running Armory with the --interactive flag, you can interactively view and/or save off data examples.
Please see docker.md for instructions on how to run Armory interactively and running_armory_scenarios_interactively.ipynb for a tutorial which includes instructions for exporting samples. Once you’ve attached
to the container, please see the following code snippets for an example of how to view and/or save off data examples:
First, we’ll simply load our scenario config and evaluate on one batch of data:
>>> from armory.scenarios.main import get as get_scenario
>>> s = get_scenario("/armory/tmp/2022-03-18T163008.249437/interactive-config.json").load() # load config
>>> s.next() # load batch of data
>>> s.evaluate_current() # make benign prediction, generate adversarial sample, make adversarial prediction
Now at this point, let’s say you’d like to save off the benign and adversarial example:
>>> s.x.shape
(1, 32, 32, 3)
>>> s.sample_exporter.export(s.x[0], "benign_x")
>>> s.sample_exporter.export(s.x_adv[0], "adversarial_x")
After calling this method, the images are saved to the scenario output directory:
ls ~/.armory/outputs/2022-03-18T163008.249437/saved_samples/
adversarial_x.png benign_x.png
If, instead of writing to disk, you’d like to return an individual sample, use the get_sample() method:
>>> img = s.sample_exporter.get_sample(s.x_adv[0])
>>> type(img)
<class 'PIL.Image.Image'>
For object detection and video tracking scenarios, Armory exports a set of raw images as well as images containing ground-truth (red) and predicted (white) bounding boxes. This will all occur automatically if using Armory in the normal/non-interactive mode, but we include the following interactive example as well:
In the example below, we’ve already loaded an xView object detection scenario and run evaluate_current() for a single batch before running the following to export
the adversarial image with ground-truth and predicted boxes overlaid:
>>> s.sample_exporter.export(
s.x_adv[0],
"adversarial_x_with_boxes",
y=s.y[0],
y_pred=s.y_pred_adv[0],
with_boxes=True)
ls ~/.armory/outputs/2022-03-18T181736.925088/saved_samples/
adversarial_x_with_boxes.png
As depicted earlier, the get_sample() method can be used to return the PIL image. The boolean with_boxes kwarg can be used to add
bounding boxes to the image. When this is set to True, you must provide values for at least one of y and y_pred.
>>> adv_img = s.sample_exporter.get_sample(s.x_adv[0])
>>> type(adv_img)
<class 'PIL.Image.Image'>
>>> adv_img_with_boxes = s.sample_exporter.get_sample(s.x_adv[0], with_boxes=True, y=s.y[0], y_pred=s.y_pred_adv[0])
>>> type(adv_img_with_boxes)
<class 'PIL.Image.Image'>
>>> s.sample_exporter.get_sample(s.x[0], with_boxes=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/workspace/armory/utils/export.py", line 205, in get_sample
raise TypeError("Both y and y_pred are None, but with_boxes is True")
TypeError: Both y and y_pred are None, but with_boxes is True
If you’d like to include only ground-truth boxes (or only predicted boxes), provide an arg for only y (or only y_pred).
For the multimodal CARLA scenario, depth images are automatically outputted in addition to RGB when "export_batches" is set in the config. If you’d like to interactively return the depth image,
call get_sample(x_i[..., 3:]).
The get_sample() method for the So2Sat scenario exporter takes a modality arg which must be one of {'vh', 'vv', 'eo'}. Calling export() will save off all three types of examples, which will occur automatically when running a config where "export_batches" is set.
For object detection scenarios, if "export_batches" is set in the config file, Armory will output COCO-formatted JSON
files with ground-truth and predicted bounding boxes. These are saved in the scenario output directory as
ground_truth_boxes_coco_format.json, benign_predicted_boxes_coco_format.json, and adversarial_predicted_boxes_coco_format.json. Note that
this functionality exists separately from the exporters described in this document.