MCALF: Multi-Component Atmospheric Line Fitting¶

MCALF Documentation¶

Welcome to MCALF’s documentation!

MCALF is an open-source Python package for accurately constraining velocity information from spectral imaging observations using machine learning techniques.

These pages document how the package can be interacted with. Some examples are also provided. A Documentation Index and a Module Index are available.

User Documentation¶

License

MCALF is an open-source Python package for accurately constraining velocity information from spectral imaging observations using machine learning techniques.

This software package is intended to be used by solar physicists trying to extract line-of-sight (LOS) Doppler velocity information from spectral imaging observations (Stokes I measurements) of the Sun. A ‘toolkit’ is provided that can be used to define a spectral model optimised for a particular dataset.

This package is particularly suited for extracting velocity information from spectral imaging observations where the individual spectra can contain multiple spectral components. Such multiple components are typically present when active solar phenomenon occur within an isolated region of the solar disk. Spectra within such a region will often have a large emission component superimposed on top of the underlying absorption spectral profile from the quiescent solar atmosphere.

A sample model is provided for an IBIS Ca II 8542 Å spectral imaging sunspot dataset. This dataset typically contains spectra with multiple atmospheric components and this package supports the isolation of the individual components such that velocity information can be constrained for each component. Using this sample model, as well as the separate base (template) model it is built upon, a custom model can easily be built for a specific dataset.

The custom model can be designed to take into account the spectral shape of each particular spectrum in the dataset. By training a neural network classifier using a sample of spectra from the dataset labelled with their spectral shapes, the spectral shape of any spectrum in the dataset can be found. The fitting algorithm can then be adjusted for each spectrum based on the particular spectral shape the neural network assigned it.

This package is designed to run in parallel over large data cubes, as well as in serial. As each spectrum is processed in isolation, this package scales very well across many processor cores. Numerous functions are provided to plot the results in a clearly. The MCALF API also contains many useful functions which have the potential of being integrated into other Python packages.

Installation¶

For easier package management we recommend using Miniconda (or Anaconda) and creating a new conda environment to install MCALF inside. To install MCALF using Miniconda, run the following commands in your system’s command prompt, or if you are using Windows, in the ‘Anaconda Prompt’:

$ conda config --add channels conda-forge
$ conda config --set channel_priority strict
$ conda install mcalf

MCALF is updated to the latest version by running:

$ conda update mcalf

Alternatively, you can install MCALF using pip:

$ pip install mcalf

Testing¶

First, install the package as usual, and then download the code associated with your installed MCALF version. Unzip the file and navigate to it in the terminal. Run the following command (in the same directory as setup.py) to test your installation,

$ python -m pytest --cov=mcalf

Make sure you are inside the virtual environment where it was installed.

Getting Started¶

The following examples provide the key details on how to use this package. For more details on how to use the particular classes and function, please consult the Code Reference. We plan to expand this section with more examples of this package being used.

example1: Basic usage of the package¶

`FittingIBIS.ipynb`¶

This file is an IPython Notebook containing examples of how to use the package to accomplish typical tasks.

`FittingIBIS.pro`¶

Download FittingIBIS.pro

This file is similar to FittingIBIS.ipynb file, except it written is IDL. It is not recommended to use the IDL wrapper in production, just use it to explore the code if you are familiar with IDL and not Python. If you wish to use this package, please use the Python implementation. IDL is not fully supported in the current version of the code for reasons such as, the Python tuple datatype cannot be passed from IDL to Python, resulting in certain function calls not being possible.

`config.yml`¶

Download config.yml

This is an example configuration file containing default parameters. This can be easier than setting the parameters in the code. The file follows the YAML format.

Labelling Tutorial¶

This Jupyter notebook provides a simple, semi-automated, method to produce a ground truth data set that can be used to train a neural network for use as a spectral shape classifier in the MCALF package. The following code can be adapted depending on the number of classifications that you want.

Download LabellingTutorial.ipynb

Load the required packages

[ ]:

import mcalf.models
from mcalf.utils import normalise_spectrum
import numpy as np
from astropy.io import fits
import matplotlib.pyplot as plt

Load data files

[ ]:

wavelengths = np.loadtxt('wavelengths.csv', delimiter=',')  # Original wavelengths
prefilter_response_wvscl = np.loadtxt('prefilter_response_wvscl.csv', delimiter=',')
prefilter_response_main = np.loadtxt('prefilter_response_main.csv', delimiter=',')

with fits.open('spectral_data.fits') as hdul:  # Raw spectral data
    datacube = np.asarray(hdul[0].data, dtype=np.float64)

Initialise the model that will use the labelled data

[ ]:

model = mcalf.models.IBIS8542Model(original_wavelengths=wavelengths, prefilter_ref_main=prefilter_response_main,
                                     prefilter_ref_wvscl=prefilter_response_wvscl)

Select the points to label

[ ]:

i_points, j_points = np.load('labelled_points.npy')

Select the spectra to label from the data file

[ ]:

raw_spectra = datacube[:, i_points, j_points].T

Normalise each spectrum to be in range [0, 1]

[ ]:

labelled_spectra = np.empty((len(raw_spectra), len(model.constant_wavelengths)))
for i in range(len(labelled_spectra)):
    labelled_spectra[i] = normalise_spectrum(raw_spectra[i], model=model)

Script to semi-automate the classification process¶

Type a number 0 - 4 for assign a classification to the plotted spectrum
Type 5 to skip and move on to the next spectrum
Type back to move to the previous spectrum
Type exit to give up (keeping ones already done)

The labels are present in the labels variable (-1 represents an unclassified spectrum)

[ ]:

labels = np.full(len(labelled_spectra), -1, dtype=int)
i = 0
while i < len(labelled_spectra):

    # Show the spectrum to be classified along with description
    plt.figure(figsize=(15, 10))
    plt.plot(labelled_spectra[i])
    plt.show()
    print("i = {}".format(i))
    print("absorption --- both --- emission / skip")
    print("       0    1    2    3    4         5 ")

    # Ask for user's classification
    classification = input('Type [0-4]:')

    try:  # Must be an integer
        classification_int = int(classification)
    except ValueError:
        classification_int = -1  # Try current spectrum again

    if classification == 'back':
        i -= 1  # Go back to the previous spectrum
    elif classification == 'exit':
        break  # Exit the loop, saving labels that were given
    elif 0 <= classification_int <= 4:  # Valid classification
        labels[i] = int(classification)  # Assign the classification to the spectrum
        i += 1  # Move on to the next spectrum
    elif classification_int == 5:
        i += 1  # Skip and move on to the next spectrum
    else:  # Invalid integer classification
        i += 0  # Try current spectrum again

Plot bar chart of classification populations

[ ]:

unique, counts = np.unique(labels, return_counts=True)
plt.figure()
plt.bar(unique, counts)
plt.title('Number of spectra in each classification')
plt.xlabel('Classification')
plt.ylabel('N_spectra')
plt.show()

Overplot the spectra of each classification

[ ]:

for classification in unique:
    plt.figure()
    for spectrum in labelled_spectra[labels == classification]:
        plt.plot(model.constant_wavelengths, spectrum)
    plt.title('Classification {}'.format(classification))
    plt.yticks([0, 1])
    plt.show()

Save the labelled spectra for use later

[ ]:

np.save('labelled_data.npy', labelled_spectra)
np.save('labels.npy', labels)

If you are interested in using this package in your research and would like advice on how to use this package, please contact Conor MacBride.

Contributing¶

Code of Conduct

If you find this package useful and have time to make it even better, you are very welcome to contribute to this package, regardless of how much prior experience you have. Types of ways you can contribute include, expanding the documentation with more use cases and examples, reporting bugs through the GitHub issue tracker, reviewing pull requests and the existing code, fixing bugs and implementing new features in the code.

You are encouraged to submit any bug reports and pull requests directly to the GitHub repository. If you have any questions regarding contributing to this package please contact Conor MacBride.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

Citation¶

If you have used this package in work that leads to a publication, we would be very grateful if you could acknowledge your use of this package in the main text of the publication. Please cite the Zenodo DOI for the package version you used. Please also consider integrating your code and examples into the package.

License¶

MCALF is licensed under the terms of the BSD 2-Clause license.

Code Reference¶

MCALF¶

mcalf Package¶

MCALF: Multi-Component Atmospheric Line Fitting¶

MCALF is an open-source Python package for accurately constraining velocity information from spectral imaging observations using machine learning techniques.

MCALF models¶

This sub-package contains:

Base and sample models that can be adapted and fitted to any spectral imaging dataset.
Models optimised for particular data sets that can be used directly.
Data structures for storing and exporting the fitted parameters, as well as simplifying the calculation of velocities (mcalf.models.results).

mcalf.models Package¶

Classes¶

`FitResult`(fitted_parameters, fit_info)	Class that holds the result of a fit
`FitResults`(shape, n_parameters[, time])	Class that holds multiple fit results in a way that can be easily processed
`IBIS8542Model`([stationary_line_core, …])	Class for working with IBIS 8542 Å calcium II spectral imaging observations
`ModelBase`()	Base class for spectral line model fitting.

Class Inheritance Diagram¶

Inheritance diagram of mcalf.models.results.FitResult, mcalf.models.results.FitResults, mcalf.models.ibis.IBIS8542Model, mcalf.models.base.ModelBase

mcalf.models.base Module¶

Classes¶

ModelBase()

Base class for spectral line model fitting.

Class Inheritance Diagram¶

Inheritance diagram of mcalf.models.base.ModelBase

mcalf.models.ibis Module¶

Classes¶

IBIS8542Model([stationary_line_core, …])

Class for working with IBIS 8542 Å calcium II spectral imaging observations

Class Inheritance Diagram¶

Inheritance diagram of mcalf.models.ibis.IBIS8542Model

mcalf.models.results Module¶

Classes¶

`FitResult`(fitted_parameters, fit_info)	Class that holds the result of a fit
`FitResults`(shape, n_parameters[, time])	Class that holds multiple fit results in a way that can be easily processed

MCALF profiles¶

This sub-package contains:

Functions that can be used to model the spectra.
Voigt profile with a variety of wrappers for different applications (mcalf.profiles.voigt).
Gaussian profiles and skew normal distributions (mcalf.profiles.gaussian).

mcalf.profiles Package¶

Functions¶

`double_voigt`(x, a1, b1, s1, g1, a2, b2, s2, …)	Double Voigt function with background
`double_voigt_approx`(x, a1, b1, s1, g1, a2, …)	Double Voigt function (efficient approximation) with background
`double_voigt_approx_nobg`(x, a1, b1, s1, g1, …)	Double Voigt function (efficient approximation) with no background
`double_voigt_nobg`(x, a1, b1, s1, g1, a2, b2, …)	Double Voigt function with no background
`single_gaussian`(x, a, b, c, d)	Gaussian function
`skew_normal`(x, a_a, alpha, xi, omega, d)
`skew_normal_with_gaussian`(x, a_a, alpha, xi, …)
`voigt`(x, a, b, s, g, d[, clib])	Voigt function with background
`voigt_approx`(x, a, b, s, g, d)	Voigt function (efficient approximation) with background
`voigt_approx_nobg`(x, a, b, s, g)	Voigt function (efficient approximation) with no background (Base approx.
`voigt_nobg`(x, a, b, s, g[, clib])	Voigt function with no background (Base Voigt function)

mcalf.profiles.voigt Module¶

Functions¶

`voigt_approx_nobg`(x, a, b, s, g)	Voigt function (efficient approximation) with no background (Base approx.
`voigt_approx`(x, a, b, s, g, d)	Voigt function (efficient approximation) with background
`double_voigt_approx_nobg`(x, a1, b1, s1, g1, …)	Double Voigt function (efficient approximation) with no background
`double_voigt_approx`(x, a1, b1, s1, g1, a2, …)	Double Voigt function (efficient approximation) with background
`voigt_nobg`(x, a, b, s, g[, clib])	Voigt function with no background (Base Voigt function)
`voigt`(x, a, b, s, g, d[, clib])	Voigt function with background
`double_voigt_nobg`(x, a1, b1, s1, g1, a2, b2, …)	Double Voigt function with no background
`double_voigt`(x, a1, b1, s1, g1, a2, b2, s2, …)	Double Voigt function with background

mcalf.profiles.gaussian Module¶

Functions¶

`single_gaussian`(x, a, b, c, d)	Gaussian function
`skew_normal`(x, a_a, alpha, xi, omega, d)
`skew_normal_with_gaussian`(x, a_a, alpha, xi, …)

MCALF visualisation¶

This sub-package contains:

Functions to plot the input spectrum and the fitted model (mcalf.visualisation.spec).
Functions to plot the spatial distribution and their general profile (mcalf.visualisation.classifications).
Functions to plot the velocities calculated for a spectral imaging scan (mcalf.visualisation.velocity).

mcalf.visualisation Package¶

Functions¶

`plot_averaged_class_map`(class_map[, …])	Plot an image of the time averaged classifications
`plot_class_map`(class_map[, overall_classes, …])	Plot an image of the classifications at a particular time along with bar charts of the classifications
`plot_classifications`(class_map, spectra, labels)	Plot the spectra separated into their classifications along with an example classified map.
`plot_ibis8542`(wavelengths, spectrum[, fit, …])	Plot an IBIS8542Model fit
`plot_map`(velmap[, mask, umbra_mask, …])	Plot the velocity map
`plot_spectrum`(wavelengths, spectrum[, …])	Plot a spectrum with the wavelength grid shown.

mcalf.visualisation.spec Module¶

Functions¶

`plot_ibis8542`(wavelengths, spectrum[, fit, …])	Plot an IBIS8542Model fit
`plot_spectrum`(wavelengths, spectrum[, …])	Plot a spectrum with the wavelength grid shown.

mcalf.visualisation.classifications Module¶

Functions¶

`plot_classifications`(class_map, spectra, labels)	Plot the spectra separated into their classifications along with an example classified map.
`plot_class_map`(class_map[, overall_classes, …])	Plot an image of the classifications at a particular time along with bar charts of the classifications
`plot_averaged_class_map`(class_map[, …])	Plot an image of the time averaged classifications

mcalf.visualisation.velocity Module¶

Functions¶

plot_map(velmap[, mask, umbra_mask, …])

Plot the velocity map

MCALF utils¶

This sub-package contains:

Functions for processing spectra (mcalf.utils.spec).
Functions for smoothing n-dimensional arrays (mcalf.utils.smooth).
Functions for masking the input data to limit the region computed (mcalf.utils.mask).
Miscellaneous utility functions (mcalf.utils.misc).

mcalf.utils Package¶

Functions¶

`gaussian_kern_3d`([width, sigma])	3D Gaussian kernel
`generate_sigma`(sigma_type, wavelengths, …)	Generate the default sigma profiles
`genmask`(width, height[, radius, …])	Generate a circular mask of specified size
`load_parameter`(parameter[, wl])	Load parameters from file, optionally evaluating variables from strings
`make_iter`(*args)	Returns each inputted argument, wrapping in a list if not already iterable
`moving_average`(array, width)	Boxcar moving average
`normalise_spectrum`(spectrum[, …])	Normalise an individual spectrum to have intensities in range [0, 1]
`radial_distances`(n_cols, n_rows)	Generates a 2D array of specified shape of radial distances from the centre
`reinterpolate_spectrum`(spectrum, …)	Reinterpolate the spectrum
`smooth_cube`(cube, mask, **kwargs)	Apply Gaussian smoothing to velocities

mcalf.utils.spec Module¶

Functions¶

`reinterpolate_spectrum`(spectrum, …)	Reinterpolate the spectrum
`normalise_spectrum`(spectrum[, …])	Normalise an individual spectrum to have intensities in range [0, 1]
`generate_sigma`(sigma_type, wavelengths, …)	Generate the default sigma profiles

mcalf.utils.smooth Module¶

Functions¶

`moving_average`(array, width)	Boxcar moving average
`gaussian_kern_3d`([width, sigma])	3D Gaussian kernel
`smooth_cube`(cube, mask, **kwargs)	Apply Gaussian smoothing to velocities

mcalf.utils.mask Module¶

Functions¶

`genmask`(width, height[, radius, …])	Generate a circular mask of specified size
`radial_distances`(n_cols, n_rows)	Generates a 2D array of specified shape of radial distances from the centre

mcalf.utils.misc Module¶

Functions¶

`make_iter`(*args)	Returns each inputted argument, wrapping in a list if not already iterable
`load_parameter`(parameter[, wl])	Load parameters from file, optionally evaluating variables from strings

Contributor Covenant Code of Conduct¶

Our Pledge¶

We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.

Our Standards¶

Examples of behavior that contributes to a positive environment for our community include:

Demonstrating empathy and kindness toward other people
Being respectful of differing opinions, viewpoints, and experiences
Giving and gracefully accepting constructive feedback
Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
Focusing on what is best not just for us as individuals, but for the overall community

Examples of unacceptable behavior include:

The use of sexualized language or imagery, and sexual attention or advances of any kind
Trolling, insulting or derogatory comments, and personal or political attacks
Public or private harassment
Publishing others’ private information, such as a physical or email address, without their explicit permission
Other conduct which could reasonably be considered inappropriate in a professional setting

Enforcement Responsibilities¶

Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.

Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.

Scope¶

This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.

Enforcement¶

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at mcalf@macbride.me. All complaints will be reviewed and investigated promptly and fairly.

All community leaders are obligated to respect the privacy and security of the reporter of any incident.

Enforcement Guidelines¶

Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:

1. Correction¶

Community Impact: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.

Consequence: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.

2. Warning¶

Community Impact: A violation through a single incident or series of actions.

Consequence: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.

3. Temporary Ban¶

Community Impact: A serious violation of community standards, including sustained inappropriate behavior.

Consequence: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.

4. Permanent Ban¶

Community Impact: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.

Consequence: A permanent ban from any sort of public interaction within the community.

Attribution¶

This Code of Conduct is adapted from the Contributor Covenant, version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.

Community Impact Guidelines were inspired by Mozilla’s code of conduct enforcement ladder.

For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq. Translations are available at https://www.contributor-covenant.org/translations.

MCALF Licence¶

MCALF is licensed under the terms of the BSD 2-Clause license.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

MCALF: Multi-Component Atmospheric Line Fitting¶

MCALF Documentation¶

User Documentation¶

Installation¶

Testing¶

Getting Started¶

example1: Basic usage of the package¶

FittingIBIS.ipynb¶

FittingIBIS.pro¶

config.yml¶

Labelling Tutorial¶

Script to semi-automate the classification process¶

Contributing¶

Citation¶

License¶

Code Reference¶

MCALF¶

mcalf Package¶

MCALF: Multi-Component Atmospheric Line Fitting¶

MCALF models¶

mcalf.models Package¶

Classes¶

Class Inheritance Diagram¶

mcalf.models.base Module¶

Classes¶

Class Inheritance Diagram¶

mcalf.models.ibis Module¶

Classes¶

Class Inheritance Diagram¶

mcalf.models.results Module¶

Classes¶

MCALF profiles¶

mcalf.profiles Package¶

Functions¶

mcalf.profiles.voigt Module¶

Functions¶

mcalf.profiles.gaussian Module¶

Functions¶

MCALF visualisation¶

mcalf.visualisation Package¶

Functions¶

mcalf.visualisation.spec Module¶

Functions¶

mcalf.visualisation.classifications Module¶

Functions¶

mcalf.visualisation.velocity Module¶

Functions¶

MCALF utils¶

mcalf.utils Package¶

Functions¶

mcalf.utils.spec Module¶

Functions¶

mcalf.utils.smooth Module¶

Functions¶

mcalf.utils.mask Module¶

Functions¶

mcalf.utils.misc Module¶

Functions¶

Contributor Covenant Code of Conduct¶

Our Pledge¶

Our Standards¶

Enforcement Responsibilities¶

Scope¶

Enforcement¶

Enforcement Guidelines¶

1. Correction¶

2. Warning¶

3. Temporary Ban¶

4. Permanent Ban¶

Attribution¶

MCALF Licence¶

`FittingIBIS.ipynb`¶

`FittingIBIS.pro`¶

`config.yml`¶