逐像素解讀《Clash of Clans》遊戲畫面

Clash of Clans gives you 30 seconds to look at a base before the attack starts. That's when you figure out where the loot is, where to go in, where you can even drop troops. I wanted a model that could look at that screen for me.

Couldn't find a dataset for it. So I built one.

What's actually on the screen

More than you'd think. Buildings, walls, traps, rocks, trees, decorations, troops, heroes, the army bar at the bottom, loot and timer at the top. And everything overlaps. Decorations sit in front of buildings, the UI sits on top of the base.

So every object is a polygon of only the pixels you can actually see. If a statue covers half a cannon, the cannon's mask is just the half that shows.

Classes:

Buildings with their level, because a level 10 cannon and a level 15 cannon don't look alike
Walls by shape (leaning left, leaning right, joined)
Troops, heroes, the cards in the army bar
UI and HUD text

That ends up at 1,912 classes.

Before, during, after

A base looks pretty different depending on when you grab the screen, so I kept three scenes:

Scene	Frames
Scout screen, before the battle	5,496
During the battle	1,316
End of battle	97

6,909 frames, around 4.2M masks. YOLO seg and COCO.

The val set is split by base. Same base in train and val and your model just memorises it, and your numbers lie to you. None of the val bases are in training.

First thing I trained on it

A deploy zone model. Every base has a red border you can't drop troops inside, and on crowded bases it's honestly hard to see.

The model predicts the no-deploy area as a mask and I trace the border from that. Plain U-Net, two sizes: MobileNetV2 (27 MB) and EfficientNet-B4 (78 MB). The bigger one also learns the UI as its own class so the mask stops leaking onto the army bar.

0.996 IoU on the no-deploy area on val.

逐像素解讀《Clash of Clans》遊戲畫面

What's actually on the screen

Before, during, after

First thing I trained on it

Links