firefly/ntbks/settings_tutorial.ipynb

%load_ext autoreload
%autoreload 2
from IPython.display import YouTubeVideo

A recording of this jupyter notebook in action is available at:

YouTubeVideo("Xt1vce_2DYo")

YouTubeVideo("ft0Y3XNJhl4")

import sys
import os
import numpy as np
from firefly.data_reader import ParticleGroup,Settings,ArrayReader

don't have phil's colormaps

Tutorial notebook: Managing Custom Settings

One of the core features of Firefly is the ability to customize the user interface (UI) and the startup behavior to make bespoke iterations of Firefly using ones own data. We have organized the different options that one can customize into different settings groups that fall into two categories: those that affect the app as a whole and those that are particular to an individual group of particles.

App Settings	Particle Group Settings
Startup	Startup
UI	UI
Window	Filter
Camera	Colormap

Specific information for each key can be found in this documentation.

To create the necessary JSON files one should use the firefly.data_reader.Settings class to create a Settings object. Once you have a Settings object you can manipulate the settings as you see fit and then either 1. manually save it to a file using the outputToJSON() method or 2. connect it to a firefly.data_reader.Reader object in order to link it to a specific visualization (see the reader documentation for details on how to use a Reader object).

## let's create an settings object with the default keys
settings = Settings()

## we'll print the current settings to the console, organized into groups
##  (but we'll use the values=False keyword argument because we only want to see the keys for now)
settings.printKeys(values=False)

--  Settings: startup settings --
['maxVrange', 'startFly', 'friction', 'stereo', 'stereoSep', 'decimate', 'start_tween', 'CDmin', 'CDmax', 'CDlognorm', 'columnDensity']

--  Settings: UI settings --
['UI', 'UIfullscreen', 'UIsnapshot', 'UIreset', 'UIsavePreset', 'UIloadNewData', 'UIcameraControls', 'UIdecimation']

--  Settings: window settings --
['title', 'loaded', 'annotation', 'showFPS', 'showMemoryUsage', 'memoryLimit']

--  Settings: camera settings --
['center', 'camera', 'cameraRotation', 'cameraUp', 'quaternion']

--  Settings: particle startup settings --
['plotNmax', 'color', 'sizeMult', 'showParts', 'radiusVariable']

--  Settings: particle UI settings --
['UIparticle', 'UIdropdown', 'UIcolorPicker']

--  Settings: particle velocity settings --
['showVel', 'velType', 'velVectorWidth', 'velGradient', 'animateVel', 'animateVelDt', 'animateVelTmax']

--  Settings: particle filter settings --
['filterLims', 'filterVals', 'invertFilter']

--  Settings: particle colormap settings --
['colormapLims', 'colormapVals', 'colormap', 'colormapVariable', 'showColormap']

Settings can be changed the same way you would change a key in a dictionary

There is key validation (so you can’t attempt to set a setting that doesn’t exist) but there is no value validation, so be careful that you use appropriate values or your app might not work. See the settings documentation for details on what values each setting can take.

## let's change the title that shows up in the browser's tab list
print("before:")
## print only the settings that have to do with the window
settings.printKeys(pattern='window')
## update the title using dictionary syntax
settings['title']='---> My Favorite Data <--- '
print("after:")
## print only the settings that have to do with the window to confirm it changed
settings.printKeys(pattern='window')

before:
--  Settings: window settings --
title Firefly
loaded True
annotation None
showFPS True
showMemoryUsage True
memoryLimit 2000000000.0

after:
--  Settings: window settings --
title ---> My Favorite Data <---
loaded True
annotation None
showFPS True
showMemoryUsage True
memoryLimit 2000000000.0

Settings are most useful when connected to a firefly.data_reader.Reader object

Doing so allows many of the necessary settings to be automatically generated as additional particle groups are added.

## let's create some sample data, a grid of points in a 3d cube
my_coords = np.linspace(-10,10,20)
xs,ys,zs = np.meshgrid(my_coords,my_coords,my_coords)
xs,ys,zs = xs.flatten(),ys.flatten(),zs.flatten()
coords = np.array([xs,ys,zs]).T

## we'll pick some random field values to demonstrate filtering/colormapping
fields = np.random.random(size=xs.size)

Before we’ve attached the Settings object the particle settings are all empty.

settings.printKeys(pattern='particle')

--  Settings: particle startup settings --
plotNmax {}
color {}
sizeMult {}
showParts {}
radiusVariable {}

--  Settings: particle UI settings --
UIparticle {}
UIdropdown {}
UIcolorPicker {}

--  Settings: particle velocity settings --
showVel {}
velType {}
velVectorWidth {}
velGradient {}
animateVel {}
animateVelDt {}
animateVelTmax {}

--  Settings: particle filter settings --
filterLims {}
filterVals {}
invertFilter {}

--  Settings: particle colormap settings --
colormapLims {}
colormapVals {}
colormap {}
colormapVariable {}
showColormap {}

We’ll use a firefly.data_reader.ArrayReader, a workhorse firefly.data_reader.Reader sub-class with many convenient functions. See the reader documentation for details that are outside the scope of this tutorial.

## initialize an ArrayReader
reader = ArrayReader(
    coordinates=[coords[:-1],coords], ## pass in two particle groups as a demonstration (just copies of our sample data)
    write_to_disk=False,
    settings=settings, ## the settings object to link
    fields=[[],[fields,fields]]) ## field data for each particle group, 0 fields for 1 and 2 repeated fields for the other.

JSONdir is None, defaulting to /Users/agurvich/research/repos/Firefly/src/firefly/static/data/Data
Make sure each field_array (2) has a field_filter_flag (0), assuming True.
Make sure each field_array (2) has a field_colormap_flag (0), assuming True.
Make sure each field_array (2) has a field_radius_flag (0), assuming False.
Outputting: PGroup_0 - 7999/7999 particles - 0 tracked fields
Outputting: PGroup_1 - 8000/8000 particles - 2 tracked fields

/opt/homebrew/Caskroom/miniconda/base/envs/python38/lib/python3.9/site-packages/numpy/core/fromnumeric.py:1970: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  result = asarray(a).shape
/opt/homebrew/Caskroom/miniconda/base/envs/python38/lib/python3.9/site-packages/numpy/core/fromnumeric.py:43: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  result = getattr(asarray(obj), method)(*args, **kwds)

The original Settings object is stored in reader.settings.

## demonstrate that reader.settings is the original settings object
print('(reader.settings is settings) =',reader.settings is settings)
print()

reader.settings.printKeys(pattern='particle')

(reader.settings is settings) = True

--  Settings: particle startup settings --
plotNmax {'PGroup_0': None, 'PGroup_1': None}
color {'PGroup_0': array([0.88100143, 0.73655243, 0.77324641, 1.        ]), 'PGroup_1': array([0.36571498, 0.50979664, 0.02474504, 1.        ])}
sizeMult {'PGroup_0': 0.1, 'PGroup_1': 0.1}
showParts {'PGroup_0': True, 'PGroup_1': True}
radiusVariable {'PGroup_0': 0, 'PGroup_1': 0}

--  Settings: particle UI settings --
UIparticle {'PGroup_0': True, 'PGroup_1': True}
UIdropdown {'PGroup_0': True, 'PGroup_1': True}
UIcolorPicker {'PGroup_0': True, 'PGroup_1': True}

--  Settings: particle velocity settings --
showVel {'PGroup_0': None, 'PGroup_1': None}
velType {'PGroup_0': None, 'PGroup_1': None}
velVectorWidth {'PGroup_0': None, 'PGroup_1': None}
velGradient {'PGroup_0': None, 'PGroup_1': None}
animateVel {'PGroup_0': None, 'PGroup_1': None}
animateVelDt {'PGroup_0': None, 'PGroup_1': None}
animateVelTmax {'PGroup_0': None, 'PGroup_1': None}

--  Settings: particle filter settings --
filterLims {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}}
filterVals {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}}
invertFilter {'PGroup_0': {}, 'PGroup_1': {}}

--  Settings: particle colormap settings --
colormapLims {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}}
colormapVals {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}}
colormap {'PGroup_0': 0.015625, 'PGroup_1': 0.015625}
colormapVariable {'PGroup_0': None, 'PGroup_1': None}
showColormap {'PGroup_0': None, 'PGroup_1': None}

Notice that the dictionaries are filled with keys corresponding to each of the particle groups we passed in and sensible default values for each. The values of nested dictionaries should be changed by accessing each in turn, e.g.

settings['colormapLims']['PGroup_1']['field0'] = [0,1]

for the purposes of this tutorial, we’ll just go ahead and output the Settings object we have manually

## output the example settings file to a .json in this directory
settings.outputToJSON('.','example')

You will need to add this settings filename to filenames.json if this was not called by a Reader instance.

('./exampleSettings.json', './exampleSettings.json')

Settings can also be imported from .json files

Only settings defined in the file will be overwritten, so you can also mix-and-match settings files.

## initialize a new settings object
new_settings = Settings()

## import the settings from what we just saved above; prints the settings that are updated
new_settings.loadFromJSON("./exampleSettings.json")

replacing title
Firefly --> ---> My Favorite Data <---
replacing plotNmax
{} --> {'PGroup_0': None, 'PGroup_1': None}
replacing color
{} --> {'PGroup_0': [0.8810014255, 0.7365524281, 0.7732464137, 1.0], 'PGroup_1': [0.3657149758, 0.5097966367, 0.0247450409, 1.0]}
replacing sizeMult
{} --> {'PGroup_0': 0.1, 'PGroup_1': 0.1}
replacing showParts
{} --> {'PGroup_0': True, 'PGroup_1': True}
replacing radiusVariable
{} --> {'PGroup_0': 0, 'PGroup_1': 0}
replacing UIparticle
{} --> {'PGroup_0': True, 'PGroup_1': True}
replacing UIdropdown
{} --> {'PGroup_0': True, 'PGroup_1': True}
replacing UIcolorPicker
{} --> {'PGroup_0': True, 'PGroup_1': True}
replacing showVel
{} --> {'PGroup_0': None, 'PGroup_1': None}
replacing velType
{} --> {'PGroup_0': None, 'PGroup_1': None}
replacing velVectorWidth
{} --> {'PGroup_0': None, 'PGroup_1': None}
replacing velGradient
{} --> {'PGroup_0': None, 'PGroup_1': None}
replacing animateVel
{} --> {'PGroup_0': None, 'PGroup_1': None}
replacing animateVelDt
{} --> {'PGroup_0': None, 'PGroup_1': None}
replacing animateVelTmax
{} --> {'PGroup_0': None, 'PGroup_1': None}
replacing filterLims
{} --> {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}}
replacing filterVals
{} --> {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}}
replacing invertFilter
{} --> {'PGroup_0': {}, 'PGroup_1': {}}
replacing colormapLims
{} --> {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}}
replacing colormapVals
{} --> {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}}
replacing colormap
{} --> {'PGroup_0': 0.015625, 'PGroup_1': 0.015625}
replacing colormapVariable
{} --> {'PGroup_0': None, 'PGroup_1': None}
replacing showColormap
{} --> {'PGroup_0': None, 'PGroup_1': None}

Attaching a ParticleGroup to a Settings

One other thing you may want to do (perhaps in the course of building your own custom Reader sub-class) is link a firefly.data_reader.ParticleGroup object to a Settings object so that the different particle settings can be imported. ParticleGroup settings can be changed in settings_default attribute (which is just a normal python dictionary).

## create a test particle group
particleGroup = ParticleGroup('test',coords)
## update the color of this particle group *before* attaching it to a settings object
particleGroup.settings_default['color'] = [0,0,1,1]

## attach the particle group to the settings object
##  you can find the settings in the "particleGroup.attached_settings attribute"
new_settings.attachSettings(particleGroup)
print('(particleGroup.attached_settings is new_settings) =',particleGroup.attached_settings is new_settings)
print()
particleGroup.attached_settings.printKeys(pattern='particle')

(particleGroup.attached_settings is new_settings) = True

--  Settings: particle startup settings --
plotNmax {'PGroup_0': None, 'PGroup_1': None, 'test': None}
color {'PGroup_0': [0.8810014255, 0.7365524281, 0.7732464137, 1.0], 'PGroup_1': [0.3657149758, 0.5097966367, 0.0247450409, 1.0], 'test': [0, 0, 1, 1]}
sizeMult {'PGroup_0': 0.1, 'PGroup_1': 0.1, 'test': 0.1}
showParts {'PGroup_0': True, 'PGroup_1': True, 'test': True}
radiusVariable {'PGroup_0': 0, 'PGroup_1': 0, 'test': 0}

--  Settings: particle UI settings --
UIparticle {'PGroup_0': True, 'PGroup_1': True, 'test': True}
UIdropdown {'PGroup_0': True, 'PGroup_1': True, 'test': True}
UIcolorPicker {'PGroup_0': True, 'PGroup_1': True, 'test': True}

--  Settings: particle velocity settings --
showVel {'PGroup_0': None, 'PGroup_1': None, 'test': None}
velType {'PGroup_0': None, 'PGroup_1': None, 'test': None}
velVectorWidth {'PGroup_0': None, 'PGroup_1': None, 'test': None}
velGradient {'PGroup_0': None, 'PGroup_1': None, 'test': None}
animateVel {'PGroup_0': None, 'PGroup_1': None, 'test': None}
animateVelDt {'PGroup_0': None, 'PGroup_1': None, 'test': None}
animateVelTmax {'PGroup_0': None, 'PGroup_1': None, 'test': None}

--  Settings: particle filter settings --
filterLims {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}, 'test': {}}
filterVals {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}, 'test': {}}
invertFilter {'PGroup_0': {}, 'PGroup_1': {}, 'test': {}}

--  Settings: particle colormap settings --
colormapLims {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}, 'test': {}}
colormapVals {'PGroup_0': {}, 'PGroup_1': {'field0': None, 'field1': None}, 'test': {}}
colormap {'PGroup_0': 0.015625, 'PGroup_1': 0.015625, 'test': 0.015625}
colormapVariable {'PGroup_0': None, 'PGroup_1': None, 'test': None}
showColormap {'PGroup_0': None, 'PGroup_1': None, 'test': None}

Notice that the 'test' particle group now appears in the particle settings dictionaries (and in particular, note that settings['color']['test'] = [0,0,1,1].