layerstack package¶

Subpackages¶

layerstack.tests package

Submodules¶

layerstack.args module¶

layerstack.args¶

Classes that allow the specification of arguments that can be serialized to and loaded from json, as well as expressed on the command line.

copyright

2021, Alliance for Sustainable Energy, LLC

license

BSD-3

class layerstack.args.KwArgBase(description='', parser=None, choices=None, nargs=None, list_parser=None, save_parser=None)[source]¶

Bases: object

Base class for expressing json- and CLI-compatible arguments. Builds off of the patterns established by argparse.

name¶

name of the [keyword] argument

Type: str or None

description¶

optional description text for this [keyword] argument

Type: str

parser¶

function to parse the [keyword] argument value. analogous to the type argument in the argparse add_argument method

Type: callable

choices¶

list of allowable values

Type: None or list

nargs¶

analogous to the nargs argument in the argparse add_argument method

Type: None, int, str

list_parser¶

function to parse a [keyword] argument value list

Type: callable

save_parser¶

function to convert a [keyword] argument value to json-serializable form

Type: callable

property nargs¶

property choices_to_save¶

property is_list¶

read-only attribute

Returns: whether this [keyword] argument is a list (True) or single value (False)
Return type: bool

get_name(cleaned=True)[source]¶

add_argument_kwargs()[source]¶

Converts KwArgBase atttributes to argparse add_argument keyword arugments. For use in constructing command-line interfaces.

Returns: kwargs for argparse add_argument call
Return type: dict

class layerstack.args.Arg(name, description='', parser=None, choices=None, nargs=None, list_parser=None, save_parser=None)[source]¶

Bases: layerstack.args.KwArgBase

Defines a required, positional argument.

Parameters

name (str) – name of the argument
description (str) – optional description text for this argument
parser (callable) – function to parse the argument value. analogous to the type argument in the argparse add_argument method
choices (None or list) – list of allowable values
nargs (None, int, str) – analogous to the nargs argument in the argparse add_argument method
list_parser (callable) – function to parse an argument value list
save_parser (callable) – function to convert a [keyword] argument value to json-serializable form

property set¶

read-only attribute

Returns: whether the value has been set
Return type: bool

property value¶

settable attribute. setter parses value using parser and list_parser as appropriate

Returns: the value to which this argument has been set
Return type: any

property value_to_save¶: JSON-serializable form of the argument value. Uses save_parser if provided.

class layerstack.args.Kwarg(default=None, description='', parser=None, choices=None, nargs=None, list_parser=None, action=None, save_parser=None)[source]¶

Bases: layerstack.args.KwArgBase

Defines an optional or defaulted keyword argument.

default¶

default value to be used if no explicit value has been set. is parsed with parser and list_parser as appropriate, if default is not None

Type: None or any

description¶

optional description text for this [keyword] argument

Type: str

parser¶

function to parse the [keyword] argument value. analogous to the type argument in the argparse add_argument method

Type: callable

choices¶

list of allowable values

Type: None or list

nargs¶

analogous to the nargs argument in the argparse add_argument method

Type: None, int, str

list_parser¶

function to parse a [keyword] argument value list

Type: callable

action¶

analogous to the action argument in the argparse add_argument method. allows, e.g., command-line boolean flags.

Type: str

save_parser¶

function to convert a [keyword] argument value to json-serializable form

Type: callable

property defaulted¶

read-only attribute

Returns: whether a non-default value has been set
Return type: bool

property value¶

settable attribute. setter parses value using parser and list_parser as appropriate. setting to None clears any user-defined value and reverts to the default.

Returns: the value to which this argument has been set
Return type: any

property value_to_save¶: JSON-serializable form of the keyword argument value. Uses save_parser if provided and self.value is not None.

property default_to_save¶: JSON-serializable form of the keyword argument default value. Uses save_parser if provided and self.default is not None.

add_argument_kwargs()[source]¶

Converts KwArgBase atttributes to argparse add_argument keyword arugments. For use in constructing command-line interfaces.

Returns: kwargs for argparse add_argument call
Return type: dict

class layerstack.args.ArgMode(value)[source]¶

Bases: enum.Enum

Enumeration describing whether the KwArgs in an ArgList or KwargDict are currently being DESCribed (that is, they are being added to or removed from the container objects, or are generally being edited), or USEd (that is, the values are being set or queried in preparation for running a Layer or a Stack).

DESC = 1¶

USE = 2¶

class layerstack.args.ArgList(iterable=[], mode=ArgMode.DESC)[source]¶

Bases: list

List that contains Arg objects. In describe mode (mode == ArgMode.DESC), Arg objects may be added to and removed from this list. Their descriptions and other attributes may be edited. In use mode (mode == ArgMode.USE), only the argument values are accessible through this class’s get, set, and iter methods.

Parameters

iterable (list or other iterable) – passed to list.__init__. if non-empty, must contain Arg objects.
mode (ArgMode) – mode for this ArgList to be in upon return

property mode¶

settable attribute

Returns: current mode of this ArgList, that is, whether it is being used to DESCribe or USE arguments
Return type: ArgMode

add_arguments(parser)[source]¶

Adds this ArgList’s arguments to parser, for use in a command-line interface.

Parameters: parser (argparse.ArgumentParser) – parser.add_argument is called once for each Arg in this ArgList
Raises: LayerStackRuntimeError – if mode != ArgMode.DESC, since representing this ArgList’s arguments in an argparse.ArgumentParser is a descriptive task

set_args(cli_args)[source]¶

Sets this ArgList’s argument values to those in cli_args, which is assumed to have come from argparse.

Parameters

cli_args (argparse.Namespace) – object returned by parser.parse_args(), where parser is an argparse.ArgumentParser to which this ArgList’s add_arguments method was applied

Raises

LayerStackRuntimeError – if mode != ArgMode.USE, since setting this ArgList’s argument values is a form of using a fixed ArgList
LayerStackRuntimeError – if an expected argument name is not found in cli_args
Exception – there could be a failure in the value parsing process. value and value list parsers are provided by users, so there could be unexpected behavior.

property set¶

read-only attribute. behavior is independent of mode

Returns: whether all of the Args in this ArgList have had their values set
Return type: bool

property names¶

read-only attribute. behavior is independent of mode

Returns: list of the names of the Args in this ArgList
Return type: list of str

class layerstack.args.KwargDict(iterable=[], mode=ArgMode.DESC)[source]¶

Bases: collections.OrderedDict

OrderedDict that contains Kwarg objects. In describe mode (mode == ArgMode.DESC), Kwarg objects may be add and removed. Their descriptions and other attributes may be edited. In use mode (mode == ArgMode.USE), only the argument values are accessible through this class’s get, set, and iter methods.

Parameters

iterable (list or other iterable) – passed to collections.OrderedDict.__init__. if non-empty, values must be Kwarg objects.
mode (ArgMode) – mode for this KwargDict to be in upon return

property mode¶

settable attribute

Returns: current mode of this KwargDict, that is, whether it is being used to DESCribe or USE arguments
Return type: ArgMode

items()[source]¶

return the items in this KwargDict

Returns: name is always the Kwarg.name. the value is either the Kwarg itself (if mode == ArgMode.DESC), or its .value (if mode == ArgMode.USE)
Return type: list of name, value pairs

iteritems()[source]¶

iterate throught the items in this KwargDict

Yields: name, value pair – name is always the next Kwarg.name. the value is either the next Kwarg itself (if mode == ArgMode.DESC), or its .value (if mode == ArgMode.USE)

values()[source]¶

return the list of values in this KwargDict

Returns: values are either Kwargs (if mode == ArgMode.DESC) or the Kwarg.values (if mode == ArgMode.USE)
Return type: list of values

itervalues()[source]¶

iterate through the list of values in this KwargDict

Yields: value – value is either the next Kwarg (if mode == ArgMode.DESC) or the next Kwarg.value (if mode == ArgMode.USE)

add_arguments(parser, short_names=[])[source]¶

Adds this KwargDict’s keyword arguments to parser, for use in a command-line interface.

Parameters

parser (argparse.ArgumentParser) – parser.add_argument is called once for each Arg in this ArgList
short_names (list) – list of short names (e.g., ‘d’ for an argument specified as ‘-d’, ‘–debug’) already registered in parser

Raises

LayerStackRuntimeError – if mode != ArgMode.DESC, since representing this KwargDict’s keyword arguments in an argparse.ArgumentParser is a descriptive task

set_kwargs(cli_args)[source]¶

Sets this KwargDict’s keyword argument values to those in cli_args, which is assumed to have come from argparse.

Parameters

cli_args (argparse.Namespace) – object returned by parser.parse_args(), where parser is an argparse.ArgumentParser to which this ArgList’s add_arguments method was applied

Raises

LayerStackRuntimeError – if mode != ArgMode.USE, since setting this ArgList’s argument values is a form of using a fixed ArgList
LayerStackRuntimeError – if an expected argument name is not found in cli_args
Exception – there could be a failure in the value parsing process. value and value list parsers are provided by users, so there could be unexpected behavior.

layerstack.args.get_short_name(name, short_names=[], seps=['_', '-'])[source]¶

layerstack.layer module¶

copyright

2021, Alliance for Sustainable Energy, LLC

license

BSD-3

class layerstack.layer.LayerBase[source]¶

Bases: object

Abstract base class for user-defined layers. All attributes and methods are class level.

name = None¶

layer name, expected to be human-readable (spaces are okay and even preferred)

Type: str

uuid = None¶

unique identifier for the layer

Type: uuid.uuid4

version = None¶

human readable version number for the layer (defaults to ‘0.1.0’)

Type: str

desc = None¶

layer description

Type: str

classmethod args(**kwargs)[source]¶

Each layer must define its positional arguments by populating and returning an ArgList object.

Returns: ArgList object describing the layer’s positional arguments. Arg names should appear as positional arguments in the apply method in the same order as they are defined here.
Return type: ArgList

classmethod kwargs(**kwargs)[source]¶

Each layer must define its keyword arguments by populating and returning a KwargDict object.

Returns: KwargDict object describing the layer’s keyword arguments. Keyword argument specifications in the apply method should match what is defined in this method (i.e., be equivalent to Kwarg.name=Kwarg.default).
Return type: KwargDict

classmethod apply(stack, *args, **kwargs)[source]¶

Run this layer in the context of the stack, with positional and keyword arguments. In general in user-defined layers (classes derived from LayerBase), *args and **kwargs should be replaced by the actual positional argument names (defined in the args method) and keyword argument name, default value pairs (defined in the kwargs method).

Parameters

stack (layerstack.stack.Stack) – Stack class instance in which the layer is being run
*args – the layer’s positional arguments
**kwargs – the layer’s keyword arguments

classmethod main(log_format='%(asctime)s|%(levelname)s|%(name)s|\n\t%(message)s')[source]¶

Single-layer command-line interface entry point.

Parameters: log_format (str) – custom logging format to use with the logging package via layerstack.start_console_log

class layerstack.layer.ModelLayerBase[source]¶

Bases: layerstack.layer.LayerBase

Abstract base class for user-defined layers that operate on a model.

classmethod _check_model_type(model)[source]¶

Check model type to ensure it is operable. Raises an exception if the model is the wrong type or some other error is encountered. Derived classes should implement this method.

Parameters: model – model to be operated on
Returns
Return type: None

classmethod _load_model(model_path)[source]¶

Load model from cli. Derived classes must implement this method if they support loading from disk.

Parameters: model_path ('str') – Path from which to load model
Returns
Return type: model

classmethod _save_model(model, model_path)[source]¶

Save model from cli. Derived classes must implement this method if they support saving the model out to disk.

Parameters

model – model to be saved to disk
model_path ('str') – Path to save model to

classmethod args(model=None, **kwargs)[source]¶

Create ArgList

Parameters: model (None or a model) – model to be operated on
Returns: ArgList class instance containing list of layer’s args
Return type: ‘ArgList’

classmethod kwargs(model=None, **kwargs)[source]¶

Create KwargDict

Parameters: model – model to be operated on
Returns: KwargDict class instance containing dict of layer’s kwargs
Return type: ‘KwargDict’

classmethod apply(stack, model, *args, **kwargs)[source]¶

Run this layer in the context of the stack, with positional and keyword arguments. In general in user-defined layers (classes derived from LayerBase), *args and **kwargs should be replaced by the actual positional argument names (defined in the args method) and keyword argument name, default value pairs (defined in the kwargs method).

Parameters

stack ('Stack') – Stack class instance in which the layer is being run
model – model this layer will operate on
*args – The layer’s positional arguments
**kwargs – The layer’s keyword arguments

Returns

Updated model

Return type

model

class layerstack.layer.Layer(layer_dir, model=None)[source]¶

Bases: object

Base class to interact with layers: - Create - Load - Run

layer_dir¶

Directory containing the layer

Type: ‘str’

_layer¶

Layer object

Type: ‘LayerBase|ModelLayerBase’

_checksum¶

Checksum for layer

Type: ‘str’

_name¶

Layer name

Type: ‘str’

_args¶

Layer’s args

Type: ‘ArgList’

_kwargs¶

Layer’s kwargs

Type: ‘KwargDict’

Parameters: layer_dir ('str') – Directory from which to load the layer

classmethod create(name, parent_dir, desc=None, layer_base_class=<class 'layerstack.layer.LayerBase'>)[source]¶

Create new layer

Parameters

name (str) – Layer name
parent_dir (str or pathlib.Path) – Parent directory for layer
desc (str) – Layer description
layer_base_class (LayerBase or child class) – Base class on which to build layer

Returns

dir_path – Directory containing new layer

Return type

‘str’

static layer_filename(layer_dir)[source]¶

Create file path for layer

Parameters: layer_dir (str) – Parent directory for layer
Returns: Path to layer.py file
Return type: str

static load_layer(layer_dir)[source]¶

Load layer

Parameters: layer_dir ('str') – Parent directory for layer
Returns: Layer class object
Return type: ‘Layer’

property name¶

Get layer name

Returns: layer name
Return type: ‘str’

property layer¶

Get layer class

Returns: layer class object
Return type: ‘Layer’

property checksum¶

Get layer checksum

Returns: layer checksum
Return type: ‘str’

property args¶

Get layer args

Returns: Layer ArgList
Return type: ‘ArgList’

property kwargs¶

Get layer kwargs

Returns: Layer KwargDict
Return type: ‘KwargDict’

set_arg_mode(arg_mode)[source]¶: Convenience function for setting args and kwargs objects to the same ArgMode.

classmethod run(layer_dir, stack, model, *args, **kwargs)[source]¶

Run layer

Parameters

layer_dir ('str') – Parent directory for layer
stack ('Stack') – Stack class instance that is handling the layer
model – Model to be operated on
*args – layer args
**kwargs – layer kwargs

Returns

Return type

updated model

property runnable¶

Check if layer is runnable (i.e. all args are set)

Returns: Layer args are set and layer can be run
Return type: ‘bool’

run_layer(stack, model=None)[source]¶

Run layer

Parameters

stack ('Stack') – Stack class instance that is handling the layer
model – Model to be operated on

Returns

Return type

updated model

layerstack.stack module¶

copyright

2021, Alliance for Sustainable Energy, LLC

license

BSD-3

class layerstack.stack.Stack(layers=[], name=None, version='v0.1.0', run_dir=None, model=None)[source]¶

Bases: collections.abc.MutableSequence

Class to create, edit, save, load, and run workflows stored as sequences of Layers.

Stacks are serialized to .json files. Loading and running them is contingent upon being able to locate the Layers in the current environment.

Stacks are run in a specified directory, but at this time provide little scaffolding as far as what contents are ultimately saved to that directory.

Parameters

layers (list of layerstack.layer.Layer) – List of layers in stack
name (str) – name of stack
version (str) – stack version number
run_dir (None, str, or pathlib.Path) – directory to run stack in
model (None, str, or other) – Model to apply layers to, if applicable. Can be None, an object, or a path to a serialized model.

name¶

stack name

Type: str

version¶

stack version

Type: str

model¶

Model to apply layers to, if applicable. Upon initialization, Stack.model may be None or a str path pointing to where a model exists that may be loaded. Stack relies on Layers that implement the ModelLayerBase interface for model loading and saving.

Type: None, str, or other

insert(i, layer)[source]¶

Insert layer into stack

Parameters

i ('int') – Position at which to insert layer
layer ('Layer') – Layer to insert into stack

append(layer)[source]¶

Append layer to end of stack

Parameters: layer ('Layer') – Layer to append to end of stack

property suggested_filename¶

Create stack filename

Returns: Stack json name
Return type: ‘str’

property uuid¶

Get stack uuid

Returns: Stack unique identifier
Return type: ‘uuid4’

property layers¶

Get layers in stack

Returns: List of layers in stack
Return type: ‘list’

property run_dir¶

run directory for the stack

Type: None or str

property runnable¶

Determine if stack is runnable: - Are args set for all layers in stack

Returns: All args are set for all layers in stack, stack can be run
Return type: ‘bool’

set_arg_mode(arg_mode)[source]¶: Convenience function for setting all args and kwargs objects to the same ArgMode.

save(filename)[source]¶

Save stack

Parameters: filename ('str') – file path to save stack to

archive(filename=None)[source]¶

Archives this stack by computing the .json checksum on the without-checksum json file, and then saving a json with the checksum data added in. Called by the run method with with default filename self.run_dir / ‘stack.archive’.

Parameters: filename ('str') – file path to save stack to

classmethod get_layer_dir(layer_dir, layer_library_dirs=[], original_preferred=True, hard_fail=False)[source]¶

Determine the best path to the layerstack.layer.Layer indicated by layer_dir, based on the user-provided options. This method may be helpful when Layers and Stacks are used by different people or on different systems.

Parameters

layer_dir (str or pathlib.Path) – Location of the layerstack.layer.Layer as originally represented
layer_library_dirs (list of str or pathlib.Path) – Location(s) where the layerstack.layer.Layer may now be found. If multiple locations are provided, it is assumed that they are listed in descending preference order.
original_preferred (bool) – If True and the original layer_dir path exists, it will be used directly. If False, the original layer_dir path will only be used if the layerstack.layer.Layer is not found in any of the locations listed in layer_library_dirs.
hard_fail (bool) – If True, raises LayerStackError if the layer cannot be located.

Returns

If the layerstack.layer.Layer is located, its path is returned and pathlib.Path.exists(). Otherwise, a warning is logged and None is returned, unless hard_fail, in which case a LayerStackError is raised instead.

Return type

pathlib.Path or None

Raises

LayerStackError – If the layerstack.layer.Layer cannot be located and hard_fail is True.

classmethod load(filename, layer_library_dir=None, original_locations_preferred=True)[source]¶

Load stack from given .json file

Parameters

filename (str or pathlib.Path) – .json file path from which to load layerstack.stack.Stack
layer_library_dir (None, str, pathlib.Path, or list of str or pathlib.Path) – If one or more paths are provided, these will be used to locate layer directories listed in the layerstack.stack.Stack json file. If multiple directories are provided, they will be checked in order.
original_locations_preferred (bool) – If True, the original locations of the layers listed in the layerstack.stack.Stack json file will be preferred and use if they exist on the current file system. If False, any location(s) passed in through layer_library_dir will be preferred and checked first

Returns

result – Instantiated Stack class instance

Return type

Stack

run(save_path=None, log_level=20, archive=True)[source]¶

Run stack

Parameters

save_path (str) – Path to which results from running stack should be saved
log_level (logging) – Level of logging to be used while running stack
archive (bool) – Archive stack before running

layerstack.stack.repoint_stack(p, layer_library_dir=None, original_locations_preferred=True, run_dir=None, model=None, outfile=None)[source]¶

Load Stack from p, update run_dir and/or model, and save to outfile or to the same folder but with the file name prepended with an underscore.

Parameters

p (str) – path to Stack .json file
layer_library_dir (None, str, pathlib.Path, or list of str or pathlib.Path) – If one or more paths are provided, these will be used to locate layer directories listed in the layerstack.stack.Stack json file. If multiple directories are provided, they will be checked in order.
original_locations_preferred (bool) – If True, the original locations of the layers listed in the layerstack.stack.Stack json file will be preferred and use if they exist on the current file system. If False, any location(s) passed in through layer_library_dir will be preferred and checked first
run_dir (str or None) – run directory to use
model (str or None) – path to starting model
outfile (str or None) – where to save modified stack. If not provided, will save the result to the same directory but with the filename prepended with an underscore

layerstack.stack.parse_args_helper(args)[source]¶

layerstack.stack.main()[source]¶

Module contents¶

layerstack¶

layerstack is a Python package for assembling workflows, especially those associated with modifying, running, and analyzing simulation models

copyright

2021, Alliance for Sustainable Energy, LLC

license

BSD-3

exception layerstack.LayerStackError[source]¶

Bases: Exception

Exception base class for this package.

exception layerstack.LayerStackTypeError[source]¶

Bases: layerstack.LayerStackError

TypeError exception class for this package.

exception layerstack.LayerStackRuntimeError[source]¶

Bases: layerstack.LayerStackError

RuntimeError exception class for this package.

layerstack.start_console_log(log_level=30, log_format='%(asctime)s|%(levelname)s|%(name)s|\n\t%(message)s')[source]¶

Starts logging to the console.

Parameters

log_level (enum) – logging package log level, i.e. logging.ERROR, logging.WARN, logging.INFO or logging.DEBUG
log_format (str) – format string to use with the logging package

Returns

console_handler

Return type

logging.StreamHandler

layerstack.start_file_log(filename, log_level=30, log_format='%(asctime)s|%(levelname)s|%(name)s|\n\t%(message)s')[source]¶

Starts logging to a file.

Parameters

filename (str) – path to the log file
log_level (enum) – logging package log level, i.e. logging.ERROR, logging.WARN, logging.INFO or logging.DEBUG
log_format (str) – format string to use with the logging package

Returns

logfile

Return type

logging.FileHandler

layerstack.end_file_log(logfile)[source]¶

layerstack.checksum(filename)[source]¶

Computes the checksum of a file.

Parameters: filename (str) – file to calculate the checksum for
Returns: checksum
Return type: str

class layerstack.TempJsonFilepath[source]¶

Bases: object

Creates a temporary json filename. Usage:

with TempJsonFilepath() as tmpjson:
    # save a json file to tmpjson
    # use the file as needed
    # when this block is exited, the json file will be deleted
    # (used in this package to calculate a json checksum)

layerstack.load_module_from_file(module_name, module_path)[source]¶

Loads a python module from the path of the corresponding file. (Adapted from https://github.com/epfl-scitas/spack/blob/af6a3556c4c861148b8e1adc2637685932f4b08a/lib/spack/llnl/util/lang.py#L595-L622)

Parameters

module_name (str) – namespace where the python module will be loaded
module_path (str) – path of the python file containing the module

Returns

A valid module object

Return type

module

Raises

ImportError – when the module can’t be loaded
FileNotFoundError – when module_path doesn’t exist

layerstack.timer_str(elapsed_seconds)[source]¶