Shape Measurements From Predicted Masks are Worse Than You Think

What happened when I recomputed BI-RADS descriptors from my own model’s output instead of the expert annotations.

I spent the last few weeks rebuilding an old project of mine on breast ultrasound. The original was a small thing I put together years ago: a network, a desktop window, a Dice score. This time I wanted the output to say something a radiologist could actually read, so I added a module that measures the lesion the way the BI-RADS lexicon describes it. Shape, orientation, margin, echo pattern, posterior acoustic features.

🔗DOI: https://doi.org/10.5281/zenodo.21826921

The measurements worked beautifully on the expert masks. Boundary roughness separated benign from malignant with an AUC of 0.984 across 647 lesions in the BUSI dataset. Solidity reached 0.974, circularity 0.891. Every descriptor came out significant at p < 0.001, and every direction matched what the clinical literature says it should.

Then I fed the same measurement code the masks my own network had predicted, and most of that went away.

The numbers

Same test images, same descriptor code. The only change is where the contour came from.

Solidity fell from an AUC of 0.969 on expert masks to 0.641 on predicted ones. Circularity fell from 0.884 to 0.594. Extent from 0.844 to 0.534. Boundary roughness, the strongest descriptor of the whole set, dropped from 0.967 to 0.771.

The correlation between the two versions of the same measurement is the part that bothered me most. For circularity it is 0.12. Whatever number the pipeline reports for the circularity of a lesion, it has almost no relationship to the circularity a human annotator would have measured on the same image.

Two descriptors survived. Posterior acoustic ratio lost 0.013 of AUC and correlates at 0.87 with its expert-mask version. Echo ratio lost 0.019 and correlates at 0.63.

The split is clean and, in hindsight, obvious. Posterior ratio and echo ratio average intensity over a region. If the contour is five pixels off, the mean brightness inside it barely moves. Circularity and solidity are functions of the contour itself. A slightly ragged boundary changes the perimeter, and perimeter is squared in the circularity formula. The error does not average out. It compounds.

So a pipeline that reports “irregular shape, microlobulated margin” from an automatic segmentation is reporting something considerably weaker than the same words derived from a hand-drawn outline, and nothing in the output says so.

Making the model say so

The fix I settled on is to make the network report its own boundary confidence alongside the measurements.

I kept a dropout layer active at inference in the bottleneck, ran fifteen stochastic forward passes per image, and took the per-pixel standard deviation across them. Then I averaged that standard deviation over a three-pixel band straddling the predicted contour. Interior uncertainty is not what matters here. A network can be completely sure a lesion is present and still be guessing about where exactly it ends, and it is the ending that every shape descriptor depends on.

On the test set, that single number sorts the cases:

Cases graded high confidence had a mean Dice of 0.865, with a standard deviation of 0.067. Moderate came in at 0.526. Low at 0.263. The Spearman correlation between boundary uncertainty and Dice is −0.735.

None of that involves the ground truth. The model is telling you how good its own segmentation is, and it is roughly right.

That makes the descriptors conditionally usable. When the boundary is stable, the shape terms mean something. When it is not, the report says the contour needs checking before anyone reads a margin descriptor off it, instead of printing “spiculated” with the same typeface and the same confidence as everything else.

There is a failure mode worth knowing about. I first ran this analysis against my Attention U-Net checkpoint and every single case came back high confidence. The variance was identically zero, because that architecture has no dropout layers, so there was nothing to sample. A gate like this is silent when it breaks, and silence looks exactly like a good result.

The architecture I proposed lost

The whole point of the rebuild was a network I called RDAU-Net: residual blocks, CBAM attention after each encoder stage, attention gates on the skip connections, an ASPP bottleneck. Trained with a combined Dice, Focal Tversky and boundary loss.

It came fourth out of five.

Attention U-Net reached a mean Dice of 0.697 on the held-out test set. Plain U-Net 0.664. RDAU-Net 0.611. Paired Wilcoxon signed-rank tests on the same images put both differences below 0.05 (p = 1.1 × 10⁻⁴ against Attention U-Net, p = 0.016 against U-Net). The minimal CNN baseline managed 0.266, which at least confirms the evaluation is measuring something.

I can tell you what I think happened. RDAU-Net has 2.43 million parameters against U-Net’s 1.95 million, and its validation curve had not flattened when the schedule ended, while both baselines had plateaued around epoch 15. It also produced the highest sensitivity of any model at 0.779, so the lower Dice comes from over-segmenting rather than from missing lesions. That is a hypothesis about convergence, not a result, and I have not yet run it long enough to find out.

What I am not going to do is quietly drop the baselines from the comparison table.

Two other things I did not expect

The multi-task version predicts malignancy from features pooled over its own predicted mask. It reaches an AUC of 0.900 and an accuracy of 0.827, which sounds fine until you look at the breakdown: sensitivity 0.469, specificity 1.000. It misses 17 of the 32 malignant cases in the test set and produces zero false positives.

For cancer detection that is exactly backwards. The ranking is good, which tells me the representation has the information. The operating point is wrong, and a 0.5 threshold on an imbalanced problem was never going to be right.

The second surprise came from threshold calibration. I had written the descriptor cut-points from the literature, then added a script to refit them on the training split. Echo ratio came back with the direction inverted. On BUSI, malignant lesions were slightly less hypoechoic relative to surrounding tissue than benign ones, which is the opposite of the standard expectation. I made the script fit each direction from data and record the disagreement rather than assert the convention, because I do not know whether this is a property of this population, this scanner, or my particular way of sampling the surrounding ring.

Where this leaves things

Everything above is one hospital, one acquisition protocol, one dataset of 780 images. The next thing worth doing is running the trained models against BUS-BRA or UDIAT without retraining, and reporting how far the numbers fall. I have written the conversion script for that and have not yet run it.

The code, the trained checkpoints, the result tables and the commands to reproduce all of it are here:

🔗GitHub: https://github.com/soheilkooklan/Tumor-from-Ultrasound-Image

If you work on medical image analysis and you have a view on the boundary-uncertainty approach, or on why the attention stack underperformed, I would like to hear it.

Shape Measurements From Predicted Masks are Worse Than You Think was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.