Analyzing the Data

After importing the data, you will likely need to process and analyze the frame set. This is done through the xanespy.xanes_frameset.XanesFrameset class. Most processing and analysis steps are provided as methods on this class, so the first step is to create a frameset object.

import xanespy as xp

# Use the same HDF file and groupname as when importing
fs = xp.XanesFrameset(filename='my_analysis.h5',
                      groupname='experiment1',
                      edge=None)

Defining an Absorbance Edge

Some steps in the analysis process require knowledge of the X-ray absorbance edge to function properly. To use these features, you’ll need to define and absorbance edge for the element in question. You can also use one of the pre-defined edges, either directly or by the shortcuts xanespy.k_edges or xanespy.l_edges.

import xanespy as xp

# Using a pre-made edge by name
fs = xp.XanesFrameset(edge=xp.k_edges['Ni'], ...)

# Using a pre-made edge by reference
fs = xp.XanesFrameset(edge=xp.edges.NCANickelKEdge(), ...)

# Defining your own edge
class CuKEdge(KEdge):
    name = 'Cu'
    E_0 = 8978.9
    shell = "K"
    pre_edge = (8940, 8970)
    post_edge = (9010, 9200)
    edge_range = (8970, 9010)
    map_range = (8970, 9010)
fs = xp.XanesFrameset(edge=CuKEdge(), ...)

Forking the Data

After long computations it can be helpful to create a copy of the dataset as a sort of checkpoint. To enable this, the xanespy.xanes_frameset.XanesFrameset class includes the fork_data_group() method: it creates a copy of the HDF group. The active set can be changed by setting the data_name attribute on a XanesFrameset() object. Most operations enabled by the XanesFrameset class are not idempotent, so starting from a clean dataset may be necessary:

import xanespy
# Select an imported hdf file to use
frameset = xanespy.XanesFrameset(hdf_filename="...")
# Make sure we're starting with clean data
frameset.data_name = "imported"

# Create a copy as a checkpoint
frameset.fork_data_group("aligned")
# Do some work that we're not sure will succeed
frameset.align_frames(passes=5)

# If the alignment doesn't work right,
# we can switch back to the original data and try again
frameset.data_name = "imported"
frameset.fork_data_group("aligned")
frameset.align_frames(passes=3)

The fork_data_group() method can be slow for large datasets. Xanespy will raise exceptions for non-sensical requests for forking: trying to copy a group onto itself, using a datagroup that doesn’t exist, etc.

Frame Alignment

In order to acquire reliable spectra, it is important that the frames be aligned properly. Thermal expansion, motor slop, sample damage and imperfect microscope alignment can all cause frames to be misaligned. It is almost always necessary to align the frames before performing any of the subsequent steps.

This is done with the align_frames() method:

import xanespy
# Select an imported hdf file to use
frameset = xanespy.XanesFrameset(hdf_filename="...")

# Run through five passes of the default phase correlation
frameset.align_frames(passes=5, plot_results=True)
Box and whisker plot of translations.

With the plot_results argument, a box and whisker plot is generated showing the distribution of corrections needed for aligning each frame. Several passes help reduce the error.

The alignments are generally done with subpixel resolution, which gives improved accuracy, but requires interpolation. To avoid problems with accumulated error, a cumulative translation matrix is kept and applied at the end to the original data. You can add your own translation manually using the stage_transformations() method. If align_frames() is called with commit=False, then the alignment parameters are added to stage_transformations() but not applied. Once all transformations are staged, the apply_transformations() method will apply the cumulative transformation matrix and (by default) save the result to disk.

If the starting alignment is particularly sporadic, a false minimum can result in an exception or a very small image that doesn’t provide useful information. In these cases, it may be necessary to first stage a template registration then perform several passes of phase correlation:

fs = XanesFrameset(hdf_filename="...")
# Eg. use the 22nd energy and a range of the image as the template
template = fs.frames()[21, 110:425, 150:450]
plt.imshow(template, cmap="gray")

fs.fork_data_group('aligned')

fs.align_frames(method="template_match", template=template, commit=False)
fs.align_frames(passes=5, commit=True)

Median Filtering

There are three options for applying a median filter, with each one having a different purpose. The larger the size of the kernel given, the longer it will take to apply the filter.

Filter When Importing

Area detectors often have some number of bad pixels, either hot pixels or dead pixels. Applying a mild median filter when using one of the importers in xanespy.importers, the raw data can fix most of these problems. Some beamline importers apply this by default. The 3D filter can also include the energy dimension, but this is not recommended since the frames haven’t been aligned yet:

import xanespy as xp

xp.import_aps32idc_file(median_filter_size=(1, 5, 5))

Filter When Aligning

When aligning frames with align_frames(), it may be helpful to apply an aggressive median filter to blur each image before registration so that noise and fine details have less impact. This 2D filter is only applied to the images in memory, so does not apply to the final result.

import xanespy as xp

fs = xp.XanesFrameset(...)
fs.align_frames(median_filter_size=(5, 5))

Filter After Aligning

Depending on the scientific question being addressed, a final median filter after aligning may be desireable. This 4D filter, applied with apply_median_filter(), provides a trade-off between temporal, spatial and energy resolutions: The larger the kernel along one dimension, the less resolution you’ll be able to see but the higher the signal-to-noise in the other dimensions.

import xanespy as xp

fs = xp.XanesFrameset(...)
fs.align_frames(...)
kernel = (3, 3, 5, 5) # (time, energy, row, col)

fs.apply_median_filter(kernel)

Masking Data

Sometimes it is necessary to mask background pixels from those which contain active material. Two masking methods have been created to differentiate these two areas. edge and contrast. If mask_type is set to None, then a blank mask object will be created.

Edge

Masking by the edge method determines if the pixel contains an edge jump (pre and post edge are defined by the edge variable in the XanesFrameset). Pixels containing edge jumps are not masked while, pixels without edge jumps are masked. Sometimes the background pixels contain partial edge jumps making this method inconsistent from sample to sample. This method works exceptionally well with minimal extra User input values when the background is at a constant value. Using the Contrast method might force Users to add extra parameters to obtain an accurate mask.

fs = XanesFrameset(hdf_filename="...")
mask = fs.frame_mask(mask_type='edge')

Contrast

Masking by the contrast method differentiates pixel contrast through a scipy.filters.threshold_otsu method. Based on the contrast difference from pixel to pixel, a mask will be created. This method is to be used for any non-static background samples. Additional parameters including ‘senstivity’, ‘min_size’, and ‘frame_idx’ should be used to achieve an accurate frame mask.

fs = XanesFrameset(hdf_filename="...")
mask = fs.frame_mask(mask_type='contrast')

Subtracting Surroundings

Sometimes there are differences in the absorbance of the whole frame, including background material. This can be removed from each frame using subtract_surroundings(), giving a better spectrum. This is more likely to be useful for full-field microscopy than scanning microscopy.

fs = XanesFrameset(hdf_filename="...")
fs.subtract_surroundings()
Spectrum showing before and after subtract_surroundings

The effect of the subtract_surroundings() method.

Calculating Maps

Several basic maps can be create with the calculate_maps() method. These maps will be saved in the HDF5 file alongside the frames.

import matplot.pyplot as plt
from xanespy import xp

fs = xp.XanesFrameset()
fs.calculate_maps()

# Visualize one of the newly created
fs.plot_map(map_name='optical_depths_mean')

More fine-grained mapping is planned and will be available soon.

Fitting Spectra

When numerical methods are insufficient, it may be necessary to fit the pixel spectra with a model function and extract parameters from the model. A comprehensive guide can be found on the page Fitting X-ray Absorbance Spectra.