QueleaGuard 如何為空間聚集的生態數據設計交叉驗證

Random k-fold cross-validation can overstate model performance on clustered ecological data. But the obvious alternatives failed on my dataset too. Here is the spatial fold design I settled on for QueleaGuard, why the first two attempts failed, and what I verified before training a model.

Before QueleaGuard trains a research model, it needs a reliable way to evaluate whether that model generalises. For ecological point data, this is harder than it sounds. I tried two validation designs that failed before settling on one I trust.

Why Random Folds Are Not an Option

Observations close together in space tend to share characteristics: similar rainfall, similar vegetation, and similar distances to water.

If a random split places one observation in the training set and its neighbour a few hundred metres away in the test set, the model is partly being evaluated on data it has effectively already seen.

Research by Roberts et al. (2017) and Valavi et al. (2019) explains why ignoring spatial structure can underestimate predictive error and why spatially structured validation is often necessary.

The question for QueleaGuard was not whether to account for spatial structure. It was how to do so with an extremely uneven dataset.

Attempt 1: Use the Grid I Already Had

The analysis already divided the area around Ahero into 328 grid cells, each approximately 5.5 km across.

Blocking observations by these cells seemed like the natural starting point. But only 93 of the 328 cells contained records, and one persistent site along the Winam Gulf shoreline accounted for 62 of the 266 observations.

Grouping folds by grid cell would have produced folds dominated by a small number of cells, while others would have contributed little or no data.

The existing grid was useful for spatial analysis, but it did not automatically provide a workable cross-validation design.

Attempt 2: Cluster All the Points

Next, I applied k-means clustering to the coordinates of all records, using five clusters.

One resulting fold contained no presence observations at all.

In hindsight, the problem was straightforward. Background points were distributed across the entire 50 km analysis extent, while presence observations were concentrated around the rice scheme and the persistent shoreline site.

Clustering all points together could isolate a region containing only background observations.

That creates an evaluation problem: a fold without positive cases cannot provide meaningful estimates of recall or F1, and precision may also be undefined depending on the model's predictions.

A spatial partition can look reasonable geographically and still be unsuitable for the evaluation task.

What Worked: Anchor the Clusters on Presences

I changed the clustering strategy.

Instead of fitting k-means to all observations, I fitted it to presence coordinates only, using five clusters and a fixed random seed. I then assigned every record—both presence and background—to its nearest centroid.

This placed the centroids among the observed presences, ensuring that every cluster contained at least one presence observation.

The resulting fold distribution was:

Fold| Presence| Background| Total
0| 8| 42| 50
1| 9| 5| 14
2| 11| 20| 31
3| 36| 22| 58
4| 69| 44| 113
Total| 133| 133| 266

The folds are uneven. I kept them that way.

Fold 4 is large because it contains the persistent shoreline site, while fold 1 is small. Forcing the folds into equal sizes would mean prioritising numerical balance over the spatial structure I was trying to preserve.

The objective was not to make every fold identical. It was to construct spatially meaningful partitions that remained usable for evaluation.

Checking That the Folds Do What They Claim

I checked the resulting assignments against several invariants.

The persistent site stays together. All 62 records from the shoreline site landed in one fold, preventing that site's observations from being split between training and testing.

Grid-cell splitting is limited. Only one grid cell was split across two folds. Its two records are 4.6 km apart, farther than the largest 2 km neighbourhood used by the rice-landscape features. Consequently, those records do not share the same feature window.

This does not eliminate every possible source of spatial autocorrelation, and I do not claim that it does.

Every record is accounted for. The fold file covers the modelling dataset exactly: 266 records, each assigned once, without duplicates, across five fold IDs. Every fold contains both classes.

The assignment is reproducible. Repeating the clustering with the same data and configuration produces zero mismatches in fold assignments.

These checks do not prove that the validation design eliminates all spatial dependence. They establish that the implementation satisfies its intended assignment rules and addresses the specific failure modes encountered in the earlier attempts.

A Related Decision: Keep Coordinates Out of the Model

The same reasoning influenced the baseline predictor set.

I deliberately excluded raw latitude and longitude.

With observations this geographically concentrated, a model given raw coordinates could learn where the clusters occur instead of learning which environmental conditions are associated with quelea occurrence.

That would complicate interpretation: even with spatially structured folds, I would need to distinguish environmental relationships from geographical patterns that the model could exploit.

The baseline instead uses ten environmental and landscape predictors, plus season, selected after checking candidate predictors for redundancy.

The intention is to evaluate relationships with environmental and landscape conditions rather than hand the model explicit coordinates as shortcuts.

What This Design Does Not Fix

Fold 1 contains only 14 observations, nine of which are presences. Metrics computed on such a small sample will be noisy.

The appropriate response is to report per-fold results alongside the aggregate results, rather than allowing one average to conceal substantial variation between folds.

The dataset itself is also small: 133 presence observations. No cross-validation design can manufacture additional independent evidence from those records.

There are further limitations. Assigning records to the nearest presence-derived centroid does not guarantee a minimum geographic separation between training and test observations, nor does it remove all spatial autocorrelation. The design should therefore be treated as a deliberate spatial partition, not proof of complete spatial independence.

Most importantly, no model has been trained using these folds yet. This work establishes the evaluation design before performance results exist.

That order matters. If the validation scheme is fixed and documented first, there is less temptation to choose whichever design produces the most favourable scores.

For QueleaGuard, the immediate achievement is not a high-performing model. It is a reproducible evaluation setup whose assumptions, trade-offs, and limitations are explicit.

Project: QueleaGuard — research into environmental, spatial, temporal, and agricultural drivers of Red-billed Quelea occurrence around the Ahero Rice Irrigation Scheme.

References

• Roberts, D. R., et al. (2017). Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography. https://doi.org/10.1111/ecog.02881
• Valavi, R., et al. (2019). blockCV: An R package for generating spatially or environmentally separated folds for k-fold cross-validation of species distribution models. Methods in Ecology and Evolution. https://doi.org/10.1111/2041-210X.13107