Importing Data into Xanespy¶
The first step in any Xanespy workflow will be to import the raw data into a common format. These importer functions are written as needed: if your preferred beamline is not here, submit an issue.
APS Beamline 32-ID-C¶
The energy_scan
script at 32-ID-C saves source data as an HDF
file. Xanespy preserves the original (“source”) file and saves
imported and processed data in a second (“desination”) HDF file to be
used in later analysis. The source file can be easily imported:
import_aps32idc_xanes_file(filename='source_xanes_example_scan_001.h5',
hdf_filename='xanespy_destination.h5',
hdf_groupname'example_scan_001',
downsample=1, square=True)
The camera at the beamline captures micrographs of 2048x2448. In most
cases this is over-kill since the focusing power of the 60 nm
zone-plate is not strong enough to take advantage of this. The extra
pixel density can be converted into an improved contrast-to-noise
ratio by the downsample
parameter. This parameter controls how
many pixels (to a power of two) are combined. In the previous example,
downsample=1
combines to \(2^1 = 2\times2\) blocks and results
in a (1024, 1224) image. downsample=2
combines \(2^2 =
4\times4\) blocks to produce a (512, 612) image.
There are sometimes corner artifacts in the camera, independent of
sample position: this can cause poor alignment of the imported
data. An easy solution is to import the data with
square=True
. This will crop the imported images to be square,
discarding the corner artifacts.
- Operando experiments generate one HDF file per energy scan. Using
- the above approach would continually overwrite the imported data,
leaving only the last scan in the import data store. To overcome
this, the
timestep
andtotal_timesteps
parameters should be used withimport_aps32idc_xanes_file()
, or the relatedimport_aps32idc_xanes_files()
function.
Option 1: Loop through the files explicitly:
# Populate a list of filenames, eg. with listdir() from the os module
filenames = []
# Loop through the files an import then one at a time
for idx, fname in enumerate(filenames):
import_aps_32idc_xanes_file(fname, hdf_filename='my_XANES_data.h5',
hdf_groupname='operando_001',
timestep=idx, total_timesteps=len(filenames))
Option 2: Convenice function.
# Populate a list of filenames, eg. with listdir() from the os module
filenames = []
# The function basically does what is shown in option 1
import_aps_32idc_xanes_files(filenames, hdf_filename='my_XANES_data.h5',
hdf_groupname='operando_002')
Any additional parameters given to the convenience function
import_aps32idc_xanes_files()
will be
passed directly to the inner
import_aps32idc_xanes_file()
function. If
using option 1, it is important that parameters controlling the data
shape are consistent across calls: total_timesteps
, append
,
downsample
, square
and exclude
.
SSRL Beamline 6-2c - Directory of XRM Files¶
In-house XANES scan scripts often save a directory full of .xrm
files with the metadata coding in the filenames. From the Xradia TXM
at SSRL beamline 6-2c, this XANES scan script can be generated with
the in-house generator, and the results can then be imported with
import_ssrl_xanes_dir()
. The list of
energies is automatically extracted from the filenames. The reference
frames will also be identified in the directory.
Example usage:
import xanespy as xp
# First a script should be created with sector_8_xanes_script()
# Once the script is done, import the data with this function
xp.import_ssrl_xanes_dir("opearando_exp1/",
hdf_filename="operando_experiments.h5")
Ptychography from 5.3.2.1 (ALS)¶
The output of the nanosurveyor reconstruction algorithm at 5.3.2.1
saves the data in h5
files. import_nanosurveyor_frameset()
copies the reconstructed images and metadata from the individual files
and combines them into a new HDF5 file for XAS analysis. The original
CCD images are left in their original HDF5 files, so they should not
be discarded.
import xanespy as xp
# This function copies the reconstructed images to a new file.
xp.import_nanosurveyor_frameset('NS_160529047/')
Given the slow nature of ptychography experiments, it may be necessary
to capture an XAS scan into multiple chunks. Passing append=True
to the importer allows datasets to be combined:
import xanespy as xp
# The first data-set is imported like normal except that the
# groupname and filename to save under are explicit.
xp.import_nanosurveyor_frameset('NS_160529047/',
hdf_filename='my_ptycho_data.h5',
hdf_groupname='my_combined_experiment')
# Now subsequent scans get the ``append=True`` argument
xp.import_nanosurveyor_frameset('NS_160529048/',
hdf_filename='my_ptycho_data.h5',
hdf_groupname='my_combined_experiment',
append=True)
xp.import_nanosurveyor_frameset('NS_160529049/',
hdf_filename='my_ptycho_data.h5',
hdf_groupname='my_combined_experiment',
append=True)
It may be necessary to only import a subset of the frames collected in
a given directory. For example, if the last frame drifted out of the
field-of-view and was re-collected in the next set of energies. The
arguments energy_range
and exclude_re
can be used to fine-tune
the list of importable files. See the documentation for
import_nanosurveyor_frameset()
for more
details.
[1] | If you’re shopping for a container format for your new data storage project, I would recommend AGAINST Microsoft OLE. This format stores data in raw binary, meaning that you need to know the encoding and structure to get meaningful data out. Instead, try HDF5: a nice open-source, well documented, type-aware format with bindings in many languages. It even plays nicely with numpy out of the box. |
Xradia Image Files (.xrm and .txrm)¶
Xradia microscopes use the Microsoft OLE container format, which is
not easily read [1]. Individual scan files are generally not that
helpful anyway. But in case you need it, there are some adapters to
.xrm
and .txrm
files, namely
xanespy.xradia.XRMFile
and
xanespy.xradia.TXRMFile
.
Note
The specification for .xrm
files is not public, so these
classes are reverse-engineered and may not be (definitely aren’t)
perfect. If you encounter problems, please submit an issue.
Opening xrm or txrm files is best done via the context manager:
import xanespy as xp
import numpy as np
# Single-image xrm file
with xp.XRMFile('my_txm_image.xrm') as f:
img = f.image_data()
assert img.ndim == 2 # (row, col)
# Multi-image txrm energy stack file
with xp.TXRMFile('my_txm_stack.txrm') as f:
# Get images one at a time by index
img = f.image_data(idx=0)
assert img.ndim == 2 # (row, col)
# Get images all at once in one big array
stack = f.image_stack()
assert stack.ndim == 3 # (prj, row, col)
assert np.array_equal(img, stack[0])
# Get X-ray energies for the images
energies = f.energies()
assert len(energies) == stack.shape[0]
The XRMFile
and
TXRMFile
classes accept an optional
flavor
keyword argument. This option affects several pieces of
metadata. See the XRMFile
documentation
for details.
APS Beamline 8-BM-B - Energy Stack (TXRM)¶
Note
The X-ray microscope that was temporarily at beamline 8-BM has been returned to NSLS-II. These functions are retained for compatibility with previously collected data.
The Xradia microscope can save an entire stack in one .txrm
file. This file can be imported using the
import_aps8bm_xanes_file()
function. The
list of energies is automatically extracted from the file. The
reference frames will then reside in a different .txrm
file.
Example usage:
import xanespy as xp
xp.import_aps_8BM_xanes_file('exp1-sample-stack.txrm',
ref_filename='exp1-reference_stack.txrm',
hdf_filename='txm-data.h5',
groupname='experiment1')
Note
Currently this function can only import one XANES stack; time-resolved measurement is not implemented. If you would find this feature valuable, please submit an issue.
APS Beamline 8-BM-B - Directory of XRM Files¶
Note
The X-ray microscope that was temporarily at beamline 8-BM has been returned to NSLS-II. These functions are retained for compatibility with previously collected data.
In-house XANES scan scripts often save a directory full of .xrm
files with the metadata coding in the filenames. From the Xradia TXM
at sector 8-BM-B, this XANES scan script can be generated with
sector8_xanes_script()
, and the results
can then be imported with
import_aps8bm_xanes_dir()
. The list of
energies is automatically extracted from the filenames. The reference
frames will also be identified in the directory.
Example usage:
import xanespy as xp
# First a script should be created with sector_8_xanes_script()
# Once the script is done, import the data with this function
xp.import_aps_8BM_xanes_dir("opearando_exp1/",
hdf_filename="operando_experiments.h5")