mantispy.metrics.diagnose_testing

mantispy.metrics.diagnose_testing(adata, groupby='Metadata_Perturbation', reference='negcon', block='Metadata_Plate', n_draws=8, alpha=0.05, seed=0, n_permutations=200, *, method='ks')

Check whether differential testing is calibrated on this screen.

Parameters:

• adata (AnnData) – Well-level profiles after the normalization and transform you plan to test with, since the results depend on both.

• groupby (str (default: 'Metadata_Perturbation')) – As in differential_features().

• reference (str | None (default: 'negcon')) – As in differential_features().

• block (str | None (default: 'Metadata_Plate')) – At well resolution, as in differential_features(). At cell resolution it names the exchangeable unit for the permutation null, normally the well; the well-level default "Metadata_Plate" is read there as the physical well, and any other column is honored so the diagnosed null equals the null your test would run.

• n_draws (int (default: 8)) – Pseudo-treatments drawn from the controls for the empirical null. More draws resolve the false positive rate better and take longer.

• alpha (float (default: 0.05)) – Nominal rate the null is compared against.

• seed (int (default: 0)) – Seed for choosing which control wells stand in for a treatment. The two hit callers’ permutation nulls are seeded by the draw index instead, so they are identical across calls that differ only in seed.

• n_permutations (int (default: 200)) – Null size for the two hit callers; smaller is faster and coarser.

• method (str (default: 'ks')) – Keyword-only. The hit_calling() method the cell-resolution checks measure, "ks" by default because the diagnostic is about pseudoreplication. "mahalanobis" whitens the distances, which divides out the between-well signal the cell-shuffle null exists to expose, so it cannot show that inflation. It is not used at well resolution, which keeps each caller’s own default.

Return type:

DataFrame

Returns:

A frame with columns check, value, expected, verdict and note, one row per check that ran, where a FAIL verdict means the check does not hold on this data. The empirical-null rows are absent when every null p-value came back non-finite, and the two hit-caller rows need at least eight reference wells.

Raises:

ValueError – At well resolution, no treatment has two wells, or there are fewer than four reference wells, leaving nothing to measure a null against. At cell resolution, the object is not stamped a known resolution (stamp it with mantispy.io.stamp(), or aggregate to wells with aggregate()); or it has no complete, replicated well column, or a block naming no obs column, to draw the well-block null from; or fewer than six reference wells, too few to leave four for the null after a two-well pseudo-treatment.

Notes

At cell resolution the checks change, because cells within a well are not independent replicates and the well-level checks describe well-level testing. Control cells are relabeled as pseudo-treatments of whole control wells and put through the two cell-resolution hit callers under two nulls. Each row’s note names the block the null draws whole, and the hit_calling rows name method too, so there is no silent substitution of the test you meant:

hit_calling well-block null rate / edistance well-block null rate

The caller drawing whole wells for its null, the design’s exchangeable unit. The count is compared against the upper tail of Binomial(n_draws, alpha) as the well-level hit-caller rows are. This is the rate to trust.

hit_calling cell-shuffle null rate / edistance cell-shuffle null rate

The same caller with the null permuting single cells instead of whole wells. Cells within a well share the well, so this shrinks the null spread by the cell count rather than the well count and runs above the well-block rate when cells are not exchangeable. The verdict compares the two: it fails when the cell-shuffle count exceeds both the well-block rate and the chance cutoff, the signature of pseudoreplication, and passes when the two rates agree.

At well resolution the checks and what each one detects:

null p < 0.05 / null p < 0.01

Control wells relabeled as treatments of the size yours have. The rate should match the nominal one. Heavy tails distort small p-values first, so a test can be calibrated at 0.05 and not at 0.01, which is closer to the range a false discovery rate works in.

null discoveries

How many of those null p-values survive Benjamini-Hochberg. A count above zero means the q-values on the real data are optimistic by roughly that much.

hit_calling null rate / edistance null rate

The same relabeling applied to the two hit callers, counted over n_draws draws. Both are permutation tests that are not fully calibrated at small control counts, so the count is compared against the upper tail of Binomial(n_draws, alpha) instead of a fixed rate. At eight draws the smallest non-zero rate is 0.125, and a threshold below that would fail a calibrated screen a third of the time. Raising n_draws sharpens the answer and moves the cutoff with it.

rank test resolution

The smallest p-value a Mann-Whitney test can return at your replication, compared with what multiple-testing correction requires. With three wells against 14 reference wells the floor is 2.9e-03 whatever the effect size, and effect_size() then silently calls nothing.

excess kurtosis

How far the features are from the normality a t-test assumes. It predicts the null checks above but is not a verdict on its own, since heavy tails matter less with enough wells per group.

wells per treatment and treatments sharing a {block} with the reference

The replicate structure the other checks depend on. A treatment whose wells share no block with the reference cannot be tested.