STATS.md

Using careless.stats

This notebook describes all the machine learning and data crossvalidation metrics that careless can compute. As an example, we describe the metrics from the analysis of anomalous signal from a thermolysin serial femtosecond crystallography dataset, as done in Hekstra, Wang et al. 2025. This notebook heavily draws on explanations from Aldama, Dalton, and Hekstra 2023, which is good prerequisite reading for this notebook; this notebook also breaks down the common validation scripts found in dw-examples.

Table of Contents

1. Elementary crystallographic statistics
2. Validation statistics
3. Machine learning metrics

Elementary crystallographic statistics

Completeness and multiplicity

careless.stats also computes standard crystallography statistics, such as completeness and $I/\sigma(I)$.

careless.completeness ./careless_stats/thermolysin_dw/thl_1p8A_grid_both.mtz -i ./careless_stats/results/completeness.png -b 10 --show > ./careless_stats/results/completeness.txt

In addition to saving the plot as a png using the -i flag, the --show flag allows interactive visualization of the plot in the matplotlib GUI. The --show flag can be applied to all subsequent plots.

cat ./careless_stats/results/completeness.txt

Resolution Range (Å)      all  non-anomalous  anomalous
             overall 0.983983       0.984726   0.983116
        34.35 - 4.01 1.000000       1.000000   1.000000
         4.01 - 3.15 1.000000       1.000000   1.000000
         3.15 - 2.73 1.000000       1.000000   1.000000
         2.73 - 2.48 1.000000       1.000000   1.000000
         2.48 - 2.29 0.999829       0.999680   1.000000
         2.29 - 2.16 1.000000       1.000000   1.000000
         2.16 - 2.04 0.999661       0.999680   0.999640
         2.04 - 1.95 0.995282       0.995543   0.994989
         1.95 - 1.87 0.977730       0.978411   0.976971
         1.87 - 1.80 0.883811       0.886589   0.880719

The completeness of the last resolution bin is 0.88, and the overall completeness is 0.984.

$\text{I}/\sigma \text{I}$

careless.isigi ./careless_stats/thermolysin_dw/thl_1p8A_grid_xval_[0-1].mtz --intensity-key I --uncertainty-key SigI -i ./careless_stats/results/isigi.png -b 10 --show

Validation statistics

$CC_\text{pred}$

Careless provides several validation statistics that may help validate the quality of the merged data. The first is the $CC_\text{pred}$, the correlation coefficient between the intensities predicted by the model and the observed intensities. A typical $CC_\text{pred}$ depends on the data quality and declines as a function of resolution bin, as is the case for most examples in Hekstra, Wang et al. 2025. The $CC_\text{pred}$ is computed for a training set and a test set ($CC_\text{pred, train}$ and ($CC_\text{pred, test}$). A large gap between the two (>0.05) indicates that there is overfitting.

careless.ccpred ./careless_stats/thermolysin_dw/thl_1p8A_grid_predictions_[0-1].mtz -o ./careless_stats/results/ccpred_res.csv \
--image ./careless_stats/results/ccpred_careless.png -b 10 -m spearman -l 0.5 1.1 --show\

The raw plot from the above shell command is reproduced below.

The train-test gap is minimal until the last resolution bin.

$CC_\text{1/2}$

Careless can provide estimates of the $CC_\text{1/2}$, the half-data-set correlation coefficient. We recommend using this measure to determine the resolution cutoff ($CC_\text{1/2}$ > 0.3). This measure is determined from the cross-validation datasets, the number of which are specified using the --half-dataset-repeats flag.

careless.cchalf ./careless_stats/thermolysin_dw/*xval_both.mtz -o ./careless_stats/results/cchalf_res.csv \
 -i ./careless_stats/results/cchalf_res.png -b 10 -m spearman -l 0. 1. --show \

$CC_\text{1/2}$ is very well-estimated by half datasets (small error bars over the half-dataset repeats) and > 0.3, even in the last resolution bin.

$CC_\text{anom}$

For anomalous datasets, we may estimate the half-dataset correlation of the anomalous difference signal.

careless.ccanom ./careless_stats/thermolysin_dw/*xval_both.mtz -o ./careless_stats/results/ccanom_res.csv \
--image ./careless_stats/results/ccanom_res.png -b 10 -l 0. 0.3 --show

The $CC_\text{anom}$ is not expected to be large, and expected to decrease as a function of resolution. We observe some high-resolution tailing in datasets merged with a bivariate prior on the input structure factors. This is due to numerical issues that have been fixed not during the processing of this dataset but by careless 0.4.9.

$R_\text{split}$

The $R_\text{split}$ is a crossvalidation metric for serial crystallography data and is described in White et al. 2012. The data are split in half and then the degree to which the half-datasets agree is quantified per resolution bin.

careless.rsplit ./careless_stats/thermolysin_dw/*xval_both.mtz -o ./careless_stats/results/rsplit_res.csv --image ./careless_stats/results/rsplit_res.png -b 10 -l 0 0.3 --show

A lower $R_\text{split}$ is better.

Image $CC_\text{pred}$

careless.image_ccpred ./careless_stats/thermolysin_dw/thl_1p8A_grid_predictions_[0-1].mtz -o ./careless_stats/results/imagecc_res.csv \
--image ./careless_stats/results/imagecc_careless.png -m weighted -l 0. 1.1 --show

The above plot displays the CCpred for each image. Images with low CCpred are either misindexed or their scales are poorly modeled.

Machine learning metrics

careless is a package for scaling and merging crystallographic data using variational inference. Every time I process a dataset using careless, I will learn the distribution of the structure factor amplitudes and the scales for each miller index, given the intensity of each observed reflection in the input dataset. The scale function is implemented as a neural network, whose parameters are optimized by minimizing an objective function ('loss').

Our objective function is the standard ELBO, whose negative we minimize: $${{{{{{{-\rm{ELBO}}}}}}}}\left(q\right)={-{\mathbb{E}}}{q}\left[\log p(I|F,{{\Sigma }})\right]+{D}{KL}\left({q}_{F}\parallel p(F)\right) $$

The first term on the right-hand side is the negative log likelihood (NLL), which represents the degree to which the structure factors faithfully represent the experimental intensities. The secod term is the Kullback-Liebler divergence between the variational distribution on the structure factor amplitudes, and a prior distribution. This term measures the similarity of two distributions that is minimal (zero) when the two are identical. We now plot the NLL, KL divergence (KLDiv), and the NLL_val, which is the NLL on a held-out validation set of reflections (the fraction of the data for which we can specify by --test-fraction. The default is --test-fraction=0.1.

careless.plot_history -s ./careless_stats/thermolysin_dw/thl_1p8A_grid_history.csv --show

This plot is interactive in the matplotlib GUI. The NLL and NLL_val are almost identical, indicating minimal overfitting, and both go down, indicating that the the neural network is learning the scales and structure factor amplitudes. The KLDiv actually goes down, as the structured prior introduced by the double-Wilson distribution is initially distinct from the factorized prior.

All these statistics can also be visualized with pandas in a python environment:

pd.read_csv("./careless_stats/thermolysin_dw/thl_1p8A_grid_history.csv"); ...