nns

nns

Module containing various simple PyTorch NN modules.

class quinn.nns.nns.Gaussian

Bases: Module

Gaussian function. \(\textrm{Gaussian}(x) = e^{-x^2}\)

__init__()

Initialization.

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor x.

Returns:

Output tensor of same size as input x.

Return type:

torch.Tensor

class quinn.nns.nns.Sine(A=1.0, T=1.0)

Bases: Module

Sine function. \(\textrm{Sin}(x) = A\sin\left(2\pi x/T\right)\)

__init__(A=1.0, T=1.0)

Initialization.

Parameters:

• A (float, optional) – Amplitude A. Defaults to 1.

• T (float, optional) – Period T. Defaults to 1.

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor x.

Returns:

Output tensor of same size as input x.

Return type:

torch.Tensor

class quinn.nns.nns.Polynomial(order)

Bases: Module

Polynomial function \(\textrm{Polynomial}(x)=\sum_{i=0}^p c_i x^i\).

order

Order of the polynomial.

Type:

int

coefs

Coefficient array of size p+1.

Type:

torch.nn.Parameter

__init__(order)

Initialization.

Parameters:

order (int) – Order of the polynomial.

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor x.

Returns:

Output tensor of same size as input x.

Return type:

torch.Tensor

class quinn.nns.nns.Polynomial3

Bases: Module

Example 3-rd order polynomial function \(\textrm{Polynomial3}(x)=a+bx+cx^2+dx^3\).

a

Constant coefficient.

Type:

torch.nn.Parameter

b

First-order coefficient.

Type:

torch.nn.Parameter

c

Second-order coefficient.

Type:

torch.nn.Parameter

d

Third-order coefficient.

Type:

torch.nn.Parameter

__init__()

Instantiate four parameters.

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor x.

Returns:

Output tensor of same size as input x.

Return type:

torch.Tensor

class quinn.nns.nns.Constant

Bases: Module

Constant function \(\textrm{Constant}(x)=C\).

constant

Constant C.

Type:

torch.nn.Parameter

__init__()

Instantiate the constant.

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor x.

Returns:

Output tensor of same size as input x.

Return type:

torch.Tensor

class quinn.nns.nns.SiLU

Bases: Module

Sigmoid Linear Unit (SiLU) function \(\textrm{SiLU}(x) = x \sigma(x) = \frac{x}{1+e^{-x}}\)

__init__()

Initialization.

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor x.

Returns:

Output tensor of same size as input x.

Return type:

torch.Tensor

class quinn.nns.nns.Expon

Bases: Module

Exponential function \(\textrm{Expon}(x) = e^{x}\)

__init__()

Initialization.

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor x.

Returns:

Output tensor of same size as input x.

Return type:

torch.Tensor

class quinn.nns.nns.TwoLayerNet(D_in, H, D_out)

Bases: Module

Example two-layer function, with a cubic polynomical between layers.

linear1

First linear layer.

Type:

torch.nn.Linear

linear2

Second linear layer.

Type:

torch.nn.Linear

cubic

Cubic layer in-between the linear ones.

Type:

torch.nn.Module

__init__(D_in, H, D_out)

Initializes give the input, output dimensions and the hidden width.

Parameters:

• D_in (int) – Input dimension \(d_{in}\).

• H (int) – Hidden layer width.

• D_out (int) – Output dimension \(d_{out}\).

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor x of size \((N,d_{in})\).

Returns:

Output tensor of size \((N,d_{out})\).

Return type:

torch.Tensor

class quinn.nns.nns.MLP_simple(hls, biasorno=True)

Bases: Module

Simple MLP example.

biasorno

Whether to use bias or not.

Type:

bool

hls

List of layer widths.

Type:

tuple[int]

indim

Input dimensionality \(d_{in}\).

Type:

int

outdim

Output dimensionality \(d_{out}\).

Type:

int

model

The PyTorch Sequential model behind the forward function.

Type:

torch.nn.Sequential

Note

Uses \(\tanh(x)\) as activation function between layers.

__init__(hls, biasorno=True)

Initialization.

Parameters:

• hls (tuple[int]) – Tuple of number of units per layer, length of list if number of layers

• biasorno (bool, optional) – Whether to use bias or not. Defaults to True.

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor x of size \((N,d_{in})\).

Returns:

Output tensor of size \((N,d_{out})\).

Return type:

torch.Tensor

nnbase

Module for the MLP NN base class.

class quinn.nns.nnbase.MLPBase(indim, outdim, device='cpu')

Bases: Module

Base class for an MLP architecture.

best_model

Best trained instance, if any.

Type:

torch.nn.Module

device

Device this object’s model will live in.

Type:

str

history

List containing training history, namely, [fepoch, loss_trn, loss_trn_full, loss_val]

Type:

list[np.ndarray]

indim

Input dimensionality.

Type:

int

outdim

Output dimensionality.

Type:

int

trained

Whether the NN is already trained.

Type:

bool

__init__(indim, outdim, device='cpu')

Initialization.

Parameters:

• indim (int) – Input dimensionality, d.

• outdim (int) – Output dimensionality, o.

• device (str) – Indicates where computations are performed and tensors are allocated. Default to ‘cpu’.

forward(x)

Forward function is not implemented in base class.

Parameters:

x (torch.Tensor) – Input of the function.

Raises:

NotImplementedError – Needs to be implemented in children.

predict(x)

Prediction of the NN.

Parameters:

x (np.ndarray) – Input array of size (N,d).

Returns:

Output array of size (N,o).

Return type:

np.ndarray

Note

Both input and outputs are numpy arrays.

Note

If trained, it uses the best trained model, otherwise it will use the current weights.

numpar()

Get the number of parameters of NN.

Returns:

Number of parameters, trainable or not.

Return type:

int

fit(xtrn, ytrn, **kwargs)

Fit function.

Parameters:

• xtrn (np.ndarray) – Input array of size (N,d).

• ytrn (np.ndarray) – Output array of size (N,o).

• **kwargs (dict) – Keyword arguments.

Returns:

Best trained instance.

Return type:

torch.nn.Module

printParams()

Print parameter names and values.

printParamNames()

Print parameter names and shapes.

predict_plot(xx_list, yy_list, labels=None, colors=None, iouts=None)

Plots the diagonal comparison figures.

Parameters:

• xx_list (list[np.ndarray]) – List of (N,d) inputs (e.g., training, validation, testing).

• yy_list (list[np.ndarray]) – List of (N,o) outputs.

• labels (list[str], optional) – List of labels. If None, set label internally.

• colors (list[str], optional) – List of colors. If None, sets colors internally.

• iouts (list[int], optional) – List of outputs to plot. If None, plot all.

Note

There is a similar function for probabilistic NN in solvers.quinn.QUiNNBase.

plot_1d_fits(xx_list, yy_list, domain=None, ngr=111, true_model=None, labels=None, colors=None)

Plotting one-dimensional slices, with the other dimensions at the nominal, of the fit.

Parameters:

• xx_list (list[np.ndarray]) – List of (N,d) inputs (e.g., training, validation, testing).

• yy_list (list[np.ndarray]) – List of (N,o) outputs.

• domain (np.ndarray, optional) – Domain of the function, (d,2) array. If None, sets it automatically based on data.

• ngr (int, optional) – Number of grid points in the 1d plot.

• true_model (callable, optional) – Optionally, plot the true function.

• labels (list[str], optional) – List of labels. If None, set label internally.

• colors (list[str], optional) – List of colors. If None, sets colors internally.

Note

There is a similar function for probabilistic NN in solvers.quinn.QUiNNBase.

nnwrap

Module for various useful wrappers to NN functions.

class quinn.nns.nnwrap.NNWrap(nnmodel)

Bases: object

Wrapper class to any PyTorch NN module to make it work as a numpy function. Basic usage is therefore \(f=NNWrap(); y=f(x)\)

indices

List containing [start index, end index) for each model parameter. Useful for flattening/unflattening of parameter arrays.

Type:

list

nnmodel

The original PyTorch NN module.

Type:

torch.nn.Module

__init__(nnmodel)

Instantiate a NN Wrapper object.

Parameters:

nnmodel (torch.nn.Module) – The original PyTorch NN module.

reinitialize_instance()

Reinitialize the underlying NN module.

__call__(x)

Calling the wrapper function.

Parameters:

x (np.ndarray) – A numpy input array of size (N,d).

Returns:

A numpy output array of size (N,o).

Return type:

np.ndarray

predict(x_in, weights)

Model prediction given new weights.

Parameters:

• x_in (np.ndarray) – A numpy input array of size (N,d).

• weights (np.ndarray) – flattened parameter vector.

Returns:

A numpy output array of size (N,o).

Return type:

np.ndarray

p_flatten()

Flattens all parameters of the underlying NN module into an array.

Returns:

A flattened (1d) torch tensor.

Return type:

torch.Tensor

p_unflatten(flat_parameter)

Fills the values of corresponding parameters given the flattened numpy form.

Parameters:

flat_parameter (np.ndarray) – A flattened form of parameters.

Returns:

List of recovered parameters, reshaped and ordered to match the model.

Return type:

list[torch.Tensor]

Note

Returning the list is secondary. The most important result is that this function internally fills the values of corresponding parameters.

calc_loss(weights, loss_fn, inputs, targets)

Calculates a given loss function with respect to model parameters.

Parameters:

• weights (np.ndarray) – weights of the model.

• loss_fn (torch.nn.Module) – pytorch loss module of signature loss(inputs, targets)

• inputs (np.ndarray) – inputs to the model.

• targets (np.ndarray) – target outputs that get compared to model outputs.

Returns:

loss of the model given the data.

Return type:

loss (float)

calc_lossgrad(weights, loss_fn, inputs, targets)

Calculates the gradients of a given loss function with respect to model parameters.

Parameters:

• weights (np.ndarray) – weights of the model.

• loss_fn (torch.nn.Module) – pytorch loss module of signature loss(inputs, targets)

• inputs (np.ndarray) – inputs to the model.

• targets (np.ndarray) – target outputs that get compared to model outputs.

Returns:

A numpy array of the loss gradient w.r.t. to the model parameters at inputs.

Return type:

np.ndarray

calc_hess_full(weigths, loss_fn, inputs, targets)

Calculates the hessian of a given loss function with respect to model parameters.

Parameters:

• weights (np.ndarray) – weights of the model.

• loss_fn (torch.nn.Module) – pytorch loss module of signature loss(inputs, targets)

• inputs (np.ndarray) – inputs to the model.

• targets (np.ndarray) – target outputs that get compared to model outputs.

Returns:

Hessian matrix of the loss with respect to the model parameters at inputs.

Return type:

np.ndarray

calc_hess_diag(weigths, loss_fn, inputs, targets)

Calculates the diagonal hessian approximation of a given loss function with respect to model parameters.

Parameters:

• weights (np.ndarray) – weights of the model.

• loss_fn (torch.nn.Module) – pytorch loss module of signature loss(inputs, targets)

• inputs (np.ndarray) – inputs to the model.

• targets (np.ndarray) – target outputs that get compared to model outputs.

Returns:

A diagonal Hessian matrix of the loss with respect to the model parameters at inputs.

Return type:

np.ndarray

class quinn.nns.nnwrap.SNet(nnmodel, indim, outdim, device='cpu')

Bases: MLPBase

A single NN wrapper of a given torch NN module. This is useful as it will inherit all the methods of MLPBase. Written in the spirit of UQ wrapper/solvers.

nnmodel

The underlying torch NN module.

Type:

torch.nn.Module

__init__(nnmodel, indim, outdim, device='cpu')

Initialization.

Parameters:

• nnmodel (torch.nn.Module) – The underlying torch NN module.

• indim (int) – Input dimensionality.

• outdim (int) – Output dimensionality.

• device (str, optional) – Device where the computations will be done. Defaults to ‘cpu’.

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor.

Returns:

Output tensor.

Return type:

torch.Tensor

quinn.nns.nnwrap.nnwrapper(x, nnmodel)

A simple numpy-ifying wrapper function to any PyTorch NN module \(f(x)= extrm{NN}(x)\).

Parameters:

• x (np.ndarray) – An input numpy array x of size (N,d).

• nnmodel (torch.nn.Module) – The underlying PyTorch NN module.

Returns:

An output numpy array of size (N,o).

Return type:

np.ndarray

quinn.nns.nnwrap.nn_surrogate(x, *otherpars)

A simple wrapper function as a surrogate to a PyTorch NN module \(f(x)=\textrm{NN}(x)\).

Parameters:

• x (np.ndarray) – An input numpy array x of size (N,d).

• otherpars (list) – List containing one element, the PyTorch NN module of interest.

Returns:

An output numpy array of size (N,o).

Return type:

np.ndarray

Note

This is effectively the same as nnwrapper. It is kept for backward compatibility.

quinn.nns.nnwrap.nn_surrogate_multi(par, *otherpars)

A simple wrapper function as a surrogate to a PyTorch NN module \(f_i(x)=\textrm{NN}_i(x)\) for i=1,…,o.

Parameters:

• x (np.ndarray) – An input numpy array x of size (N,d).

• otherpars (list[list]) – List containing one element, a list of PyTorch NN modules of interest (a total of o modules).

Returns:

An output numpy array of size (N,o).

Return type:

np.ndarray

quinn.nns.nnwrap.nn_p(p, x, *otherpars)

A NN wrapper that evaluates a given PyTorch NN module given input x and flattened parameter vector p. In other words, \(f(p,x)=\textrm{NN}_p(x).\)

Parameters:

• p (np.ndarray) – Flattened parameter (weights) vector.

• x (np.ndarray) – An input numpy array x of size (N,d).

• otherpars (list) – List containing one element, the PyTorch NN module of interest.

Returns:

A numpy output array of size (N,o).

Return type:

np.ndarray

Note

The size checks on p are missing: wherever this is used in QUiNN, the size checks are implied and correct. Use with care outside QUiNN.

mlp

class quinn.nns.mlp.MLP(indim, outdim, hls, biasorno=True, activ='relu', bnorm=False, bnlearn=True, dropout=0.0, final_transform=None, device='cpu')

Bases: MLPBase

Multilayer perceptron class.

hls

Tuple of hidden layer widths.

Type:

tuple

biasorno

Whether biases are included or not.

Type:

bool

bnorm

Whether batch normalization is implemented or not.

Type:

bool

bnlearn

Whether batch normalization is learnable or not.

Type:

bool

dropout

Dropout fraction.

Type:

float

final_transform

Final transformation. Currently only ‘exp’ is implemented for Exponential.

Type:

str

nlayers

Number of layers.

Type:

int

nnmodel

Underlying model evaluator.

Type:

torch.nn.Module

__init__(indim, outdim, hls, biasorno=True, activ='relu', bnorm=False, bnlearn=True, dropout=0.0, final_transform=None, device='cpu')

Initialization.

Parameters:

• indim (int) – Input dimensionality.

• outdim (int) – Output dimensionality.

• hls (tuple) – Tuple of hidden layer widths.

• biasorno (bool) – Whether biases are included or not.

• activ (str, optional) – Activation function. Options are ‘tanh’, ‘relu’, ‘sin’ or else identity is used.

• bnorm (bool) – Whether batch normalization is implemented or not.

• bnlearn (bool) – Whether batch normalization is learnable or not.

• dropout (float, optional) – Dropout fraction. Default is 0.0.

• final_transform (str, optional) – Final transform, if any (onle ‘exp’ is implemented). Default is None.

• device (str) – It represents where computations are performed and tensors are allocated. Default is cpu.

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor.

Returns:

Output tensor.

Return type:

torch.Tensor

rnet

Module containing ResNet class and layer weight parameterization class.

class quinn.nns.rnet.RNet(rdim, nlayers, wp_function=None, indim=None, outdim=None, biasorno=True, nonlin=True, mlp=False, layer_pre=False, layer_post=False, final_layer=None, device='cpu', init_factor=1.0, sum_dim=1)

Bases: MLPBase

Residual Neural Network (ResNet) class.

activ

Activation function.

Type:

torch.nn.Module

bias_post

Bias vector of post-resnet layer, if any.

Type:

torch.nn.Parameter

bias_pre

Bias vector of pre-resnet layer, if any.

Type:

torch.nn.Parameter

biasorno

Whether or not to include biases in each resnet layer.

Type:

bool

final_layer

If there is a final layer function. The only current option is ‘exp’ for exponential function.

Type:

str

indim

Input dimensionality d.

Type:

int

init_factor

Multiplicative factor of initialized weights.

Type:

float

layer_post

Whether there is a post-resnet linear layer.

Type:

bool

layer_pre

Whether there is a pre-resnet linear layer.

Type:

bool

mlp

If True, residual connections are ignored, and this becomes a regular MLP.

Type:

bool

nlayers

Number of layers L.

Type:

int

outdim

Output dimensionality o.

Type:

int

rdim

Width of the ResNet r, i.e. number of units in each hidden layer.

Type:

int

step_size

Time step size, 1/(L+1).

Type:

float

sum_dim

Which dimension the final sum, if any, is with respect to.

Type:

int

weight_post

Weight matrix of post-resnet layer, if any.

Type:

torch.nn.Parameter

weight_pre

Weight matrix of pre-resnet layer, if any.

Type:

torch.nn.Parameter

wp_function

Weight parameterization function.

Type:

LayerFcn

__init__(rdim, nlayers, wp_function=None, indim=None, outdim=None, biasorno=True, nonlin=True, mlp=False, layer_pre=False, layer_post=False, final_layer=None, device='cpu', init_factor=1.0, sum_dim=1)

Instantiate ResNet object.

Parameters:

• rdim (int) – Width of the ResNet r, i.e. number of units in each hidden layer.

• nlayers (int) – Number of layers L.

• wp_function (LayerFcn, optional) – Weight parameterization function. Defaults to a regular ResNet without weight parameterization.

• indim (int, optional) – Input dimensionality d. Defaults to width r.

• outdim (int, optional) – Output dimensionality o. Defaults to width r.

• biasorno (bool, optional) – Whether or not to include biases in each resnet layer. Default is True.

• nonlin (bool, optional) – Whether to use nonlinear activation function between layers. Defaults to True.

• mlp (bool, optional) – If True, residual connections are ignored, and this becomes a regular MLP. Default is False.

• layer_pre (bool, optional) – Whether there is a pre-resnet linear layer. Defaults to False.

• layer_post (bool, optional) – Whether there is a post-resnet linear layer. Defaults to False.

• final_layer (str, optional) – If there is a final layer function. Two options: “exp” for exponential function; “sum” for sum function which will reduce rank of the output tensor. Defaults to no final layer.

• device (str) – It represents where computations are performed and tensors are allocated. Default to cpu.

• init_factor (float, optional) – Multiply initial condition tensors by factor. Defaults to 1.0.

• sum_dim (int, optional) – If final layer function is sum, it will select which dimension to perform sum with respect to. Defaults to 1.

forward(x)

Forward function.

Parameters:

x (torch.Tensor) – Input tensor x of size \((N,d)\).

Returns:

Output tensor of size \((N,o)\).

Return type:

torch.Tensor

class quinn.nns.rnet.LayerFcn

Bases: object

Base class for layer weight parameterization layer functions.

npar

Number of parameters in the parameterization (parameters can be Tensors).

Type:

int

__init__()

Instantiation.

__call__(pars, t)

Call signature.

Parameters:

• pars (list[torch.nn.Parameter]) – List of parameters.

• t (float) – ‘Time’, i.e. layer number.

Raises:

NotImplementedError – Need to implement it in children.

class quinn.nns.rnet.Const

Bases: LayerFcn

Constant weight parameterization.

npar

Number of parameters. Should be 1.

Type:

int

__init__()

Instantiation.

__call__(pars, t)

Call function.

Parameters:

• pars (list[torch.nn.Parameter]) – List of parameters.

• t (float) – ‘Time’, i.e. layer number.

Returns:

Constant (independent of t).

Return type:

torch.nn.Parameter

class quinn.nns.rnet.Lin

Bases: LayerFcn

Linear weight parameterization.

npar

Number of parameters. Should be 2.

Type:

int

__init__()

Instantiation.

__call__(pars, t)

Call function.

Parameters:

• pars (list[torch.nn.Parameter]) – List of parameters.

• t (float) – ‘Time’, i.e. layer number.

Returns:

Linear in t.

Return type:

torch.nn.Parameter

class quinn.nns.rnet.Quad

Bases: LayerFcn

Quadratic weight parameterization.

npar

Number of parameters. Should be 3.

Type:

int

__init__()

Instantiation.

__call__(pars, t)

Call function.

Parameters:

• pars (list[torch.nn.Parameter]) – List of parameters.

• t (float) – ‘Time’, i.e. layer number.

Returns:

Quadratic in t.

Return type:

torch.nn.Parameter

class quinn.nns.rnet.Cubic

Bases: LayerFcn

Cubic weight parameterization.

npar

Number of parameters. Should be 4.

Type:

int

__init__()

Instantiation.

__call__(pars, t)

Call function.

Parameters:

• pars (list[torch.nn.Parameter]) – List of parameters.

• t (float) – ‘Time’, i.e. layer number.

Returns:

Cubic in t.

Return type:

torch.nn.Parameter

class quinn.nns.rnet.Poly(order)

Bases: LayerFcn

Polynomial weight parameterization.

npar

Number of parameters.

Type:

int

__init__(order)

Instantiation.

Parameters:

order (int) – Order of the polynomial.

__call__(pars, t)

Call function.

Parameters:

• pars (list[torch.nn.Parameter]) – List of parameters.

• t (float) – ‘Time’, i.e. layer number.

Returns:

Polynomial in t.

Return type:

torch.nn.Parameter

class quinn.nns.rnet.NonPar(npar)

Bases: LayerFcn

Non-parameteric weight parameterization, i.e. effectively a regular ResNet without weight parameterization.

npar

Number of parameters.

Type:

int

__init__(npar)

Instantiation.

Parameters:

npar (int) – Should be one more than the number of layers L.

__call__(pars, t)

Call function.

Parameters:

• pars (list[torch.nn.Parameter]) – List of parameters.

• t (float) – ‘Time’, i.e. layer number.

Returns:

Non-parameteric: a new parameter per t value (i.e. per layer).

Return type:

torch.nn.Parameter

losses

class quinn.nns.losses.LogLoss(nnmodel, lossparams)

Bases: Module

forward(inputs, targets)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class quinn.nns.losses.PeriodicLoss(nnmodel, lam, boundary)

Bases: Module

Example of a periodic loss regularization.

model

NN model evaluator.

Type:

callable

lam

Penalty strength.

Type:

float

bdry1

First boundary.

Type:

torch.Tensor

bdry2

Second boundary.

Type:

torch.Tensor

The loss function has a form

\[\frac{1}{N}||y_{\text{pred}}-y_{\text{target}}||^2 + \frac{\lambda}{N}||M\text{(boundary1)}-M\text{(boundary2)}||^2.\]

__init__(nnmodel, lam, boundary)

Initialization.

Parameters:

• nnmodel (torch.nn.Module) – NN model.

• lam (float, optional) – Penalty strength. Defaults to 0.

• boundary (tuple) – A tuple of form (boundary1, boundary2).

forward(inputs, targets)

Forward function.

Parameters:

• inputs (torch.Tensor) – Input tensor.

• targets (torch.Tensor) – Targets tensor.

Returns:

Loss value.

Return type:

float

class quinn.nns.losses.GradLoss(nnmodel, lam=0.0, xtrn=None, gtrn=None)

Bases: Module

Example of grad loss function, including derivative contraints.

lam

Penalty strength.

Type:

float

nnmodel

NN model evaluator.

Type:

callable

The loss function has a form

\[\frac{1}{N}||M(x_{\text{train}})-y_{\text{train}}||^2 + \frac{\lambda}{Nd}||\nabla M(x_{\text{train}})-G_{\text{train}}||_F^2.\]

__init__(nnmodel, lam=0.0, xtrn=None, gtrn=None)

Initialization.

Parameters:

• nnmodel (torch.nn.Module) – NN model.

• lam (float, optional) – Penalty strength. Defaults to 0.

• xtrn (np.ndarray, optional) – Input array of size (N,d). Needs to be user-provided: default produces assertion error.

• gtrn (np.ndarray, optional) – Gradient array of size (N,d). Needs to be user-provided: default produces assertion error.

forward(inputs, targets)

Forward function.

Parameters:

• inputs (torch.Tensor) – Input tensor.

• targets (torch.Tensor) – Target tensor.

Returns:

Loss value.

Return type:

float

class quinn.nns.losses.NegLogPost(nnmodel, fulldatasize, sigma, priorparams)

Bases: Module

Negative log-posterior loss function.

nnmodel

Model evaluator.

Type:

callable

priorparams

Dictionary of parameters of prior.

Type:

float

sigma

Likelihood data noise standard deviation.

Type:

float

fulldatasize

Full datasize. Important for weighting in case likelihood is computed on a batch.

Type:

int

pi

3.1415…

Type:

float

The negative log-posterior has the form:

\[\frac{N}{2}\log{(2\pi\sigma^2)} + \frac{1}{2\sigma^2}||M(x_{\text{train}})-y_{\text{train}}||^2 +\]

\[+\frac{N}{N_{\text{full}}} \left(\frac{1}{2\sigma_{\text{prior}}^2}||w-w_{\text{anchor}}||^2 + \frac{K}{2} \log{(2\pi\sigma_{\text{prior}}^2)}\right).\]

__init__(nnmodel, fulldatasize, sigma, priorparams)

Initialization.

Parameters:

• nnmodel (callable) – Model evaluator.

• fulldatasize (int) – Full datasize. Important for weighting in case likelihood is computed on a batch.

• sigma (float) – Likelihood data noise standard deviation.

• priorparams (float) – Dictionary of parameters of prior. If None, there will be no prior.

forward(inputs, targets)

Forward function.

Parameters:

• inputs (torch.Tensor) – Input tensor.

• targets (torch.Tensor) – Target tensor.

Returns:

Loss value.

Return type:

float

class quinn.nns.losses.NegLogPrior(sigma, anchor)

Bases: Module

Calculates a Gaussian negative log-prior.

anchor

Anchor, i.e. center vector of the gaussian prior.

Type:

torch.Tensor

sigma

The standard deviation of the gaussian prior (same for all parameters).

Type:

float

pi

3.1415..

Type:

float

The negative log-prior has the form:

\[\frac{1}{2\sigma_{\text{prior}}^2}||w-w_{\text{anchor}}||^2 + \frac{K}{2} \log{(2\pi\sigma_{\text{prior}}^2)}.\]

__init__(sigma, anchor)

Parameters:

• sigma (float) – The standard deviation of the gaussian prior (same for all parameters).

• anchor (torch.Tensor) – Anchor, i.e. center vector of the gaussian prior.

forward(model)

Forward evaluator

Parameters:

model (torch.nn.Module) – The corresponding NN module.

Returns:

Negative log-prior value.

Return type:

float

class quinn.nns.losses.CustomLoss(loss_params)

Bases: Module

Example of custom one-dimensional loss function, including derivative and periodicity contraints. Quite experimental, but a base for developing problem-specific loss functions.

model

Model evaluator.

Type:

callable

lam1

Penalty strength for the periodicity constraint.

Type:

float

lam2

Penalty strength for the derivative constraint.

Type:

float

The loss function has a form:

\[\frac{1}{N}||y_{\text{pred}}-y_{\text{target}}||^2 + \lambda_1 (M(0.5)-M(-0.5))^2 + \lambda_2 (M'(0.5)-M'(-0.5))^2\]

__init__(loss_params)

Initialization.

Parameters:

loss_params (tuple) – (model, penalty1, penalty2) pair.

forward(predictions, targets)

Forward function.

Parameters:

• predictions (torch.Tensor) – Input tensor.

• targets (torch.Tensor) – Target tensor.

Returns:

Loss value.

Return type:

float

tchutils

Various useful PyTorch related utilities.

quinn.nns.tchutils.tch(arr, device='cpu', rgrad=False)

Convert a numpy array to torch Tensor.

Parameters:

• arr (np.ndarray) – A numpy array of any size.

• device (str, optional) – It represents where tensors are allocated. Default to cpu.

• rgrad (bool, optional) – Whether to require gradient tracking or not.

Returns:

Torch tensor matching the default dtype for floating-point inputs.

Return type:

torch.Tensor

quinn.nns.tchutils.npy(arr)

Convert a torch tensor to numpy array.

Parameters:

arr (torch.Tensor) – Torch tensor of any size.

Returns:

Numpy array of the same size as the input torch tensor.

Return type:

np.ndarray

quinn.nns.tchutils.print_nnparams(nnmodel, names_only=False)

Print parameter names of a PyTorch NN module and optionally, values.

Parameters:

• nnmodel (torch.nn.Module) – The torch NN module.

• names_only (bool, optional) – Print names only. Default is False.

quinn.nns.tchutils.flatten_params(parameters)

Flattens all parameters into an array.

Parameters:

parameters (torch.nn.Parameters) – Description

Returns:

A tuple of the flattened (1d) torch tensor and a list of pairs that correspond to start/end indices of the flattened parameters.

Return type:

(torch.Tensor, list[tuple])

quinn.nns.tchutils.recover_flattened(flat_params, indices, model)

Fills the values of corresponding parameters given the flattened form.

Parameters:

• flat_params (np.ndarray) – A flattened form of parameters.

• indices (list[tuple]) – A list of pairs that correspond to start/end indices of the flattened parameters.

• model (torch.nn.Module) – The underlying PyTorch NN module.

Returns:

List of recovered parameters, reshaped and ordered to match the model.

Return type:

list[torch.Tensor]