高效深度學習的 5 篇關鍵論文分析
An analysis of 5 key papers
五篇高效深度學習論文指出，提升 AI 能力不只靠擴大模型，也可透過架構設計提高效率。ViT-Huge 在 ImageNet 達 88.5% top-1 準確率，訓練計算量比同類 CNN 少 4×，但需要 300 million 張圖片才有良好表現。Swin 以移位窗口注意力將計算複雜度由平方複雜度轉為線性複雜度。
🚀 How AI Got Smarter (Not Just Bigger): The Quiet Revolution in Efficient Deep Learning
Five papers that changed how we build AI — and why the future isn't about 100-billion-parameter models
By Rayyan · 10 min read · Published 19 oct 2026
For years, the story of AI was a story of scale.
Bigger models. More data. More GPUs. GPT-3 had 175 billion parameters. ViT-Huge had 632 million. The assumption was simple: if you want human-level AI, you need to build something enormous.
But here's what most people missed: while everyone was watching the giants, a quieter revolution was happening. A handful of researchers were asking a different question:
What if we didn't need to make models bigger — just smarter?
The answer, as it turns out, is yes. And it's changing everything.
📊 The Problem with Big
Let's be honest about what "big" actually means.
Training GPT-3 cost an estimated $4.6 million. Running it requires specialized hardware that most companies don't have. And deploying it on a phone? Forget about it.
But the real issue is deeper. Big models are:
· Expensive to train and run
· Slow at inference
· Impossible to deploy on edge devices
· Environmentally costly — training one large model emits as much CO₂ as five cars over their lifetimes
And here's the kicker: bigger doesn't always mean better. A model with 175 billion parameters can still fail at basic reasoning. Scale alone doesn't solve intelligence.
So researchers started asking: what if we could get the same performance with 10× less compute? 100× less?
That's the story of the five papers below.
┌─────────────────────────────────────────────────────────────┐
│ THE EFFICIENCY FRONTIER │
│ │
│ Accuracy │
│ ▲ │
│ │ ★ EfficientNet │
│ │ ★ │
│ │ ★ │
│ │ ★ │
│ │ ★ │
│ │ ★ │
│ │ ★ │
│ └──────────────────────────────────────────► FLOPs │
│ │
│ EfficientNet pushes the frontier: more accuracy, │
│ less compute. The gap widens at every budget level. │
└─────────────────────────────────────────────────────────────┘
🧩 Paper 1: The Transformer Goes to Vision (ViT, 2021)
For a decade, computer vision belonged to CNNs. Convolutional Neural Networks — with their sliding filters and hierarchical features — were the undisputed kings of image recognition.
Then a team at Google asked a heretical question: what if we just… didn't use convolutions at all?
Their idea, called the Vision Transformer (ViT), was almost absurdly simple. Take an image. Cut it into 16×16 patches. Treat each patch like a word in a sentence. Feed the sequence into a standard Transformer — the same architecture that powers GPT and BERT.
That's it. No convolutions. No hand-crafted features. Just patches and attention.
┌─────────────────────────────────────────────────────────────┐
│ VISION TRANSFORMER (ViT) │
│ │
│ Input Image (224×224) │
│ ┌────┬────┬────┬────┐ │
│ │ P1 │ P2 │ P3 │ P4 │ ← 16×16 patches │
│ ├────┼────┼────┼────┤ │
│ │ P5 │ P6 │ P7 │ P8 │ │
│ ├────┼────┼────┼────┤ │
│ │ P9 │P10 │P11 │P12 │ │
│ ├────┼────┼────┼────┤ │
│ │P13 │P14 │P15 │P16 │ │
│ └────┴────┴────┴────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────┐ │
│ │ Linear Projection (each patch → │ │
│ │ a vector of 768 numbers) │ │
│ └─────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────┐ │
│ │ Add [CLS] token + Positional │ │
│ │ Embeddings │ │
│ └─────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────┐ │
│ │ Transformer Encoder (12 layers) │ │
│ │ ┌───────────────────────────────┐ │ │
│ │ │ Multi-Head Self-Attention │ │ │
│ │ │ (every patch attends to │ │ │
│ │ │ every other patch) │ │ │
│ │ └───────────────────────────────┘ │ │
│ │ ┌───────────────────────────────┐ │ │
│ │ │ Multi-Layer Perceptron │ │ │
│ │ └───────────────────────────────┘ │ │
│ └─────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────┐ │
│ │ Classification Head │ │
│ │ (uses [CLS] token output) │ │
│ └─────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ "Dog" (88.5% confidence) │
└─────────────────────────────────────────────────────────────┘
The results were stunning. ViT-Huge achieved 88.5% top-1 accuracy on ImageNet — state of the art at the time — and it required 4× less compute to train than comparable CNNs.
Why it matters: ViT proved that Transformers are general-purpose. They're not just for language. They can see, too.
But there was a catch: ViT is data-hungry. It needs 300 million images to work well. On small datasets, it underperforms CNNs. The architecture had won, but it was expensive.
┌─────────────────────────────────────────────────────────────┐
│ THE DATA SCALE HYPOTHESIS │
│ │
│ Accuracy │
│ ▲ │
│ │ ● ViT │
│ │ / │
│ │ / │
│ │ / │
│ │ / │
│ │ / │
│ │ / ○ ResNet │
│ │ / / │
│ │ / / │
│ │ / / │
│ │ / / │
│ │ / / │
│ │ / / │
│ │ / / │
│ └────────/───/────────────────────────────► Data Size │
│ 1.3M 14M 300M │
│ │
│ On small data (1.3M), ResNet wins. │
│ On large data (300M), ViT overtakes dramatically. │
│ Inductive biases are a shortcut for small data. │
│ Flexible architectures win with big data. │
└─────────────────────────────────────────────────────────────┘
🧩 Paper 2: Making Transformers Efficient (Swin, 2021)
ViT had a problem: its self-attention is quadratic in image size. Double the image size, and you quadruple the compute. That makes it impractical for high-resolution images — the kind you need for object detection and segmentation.
Microsoft's Swin Transformer solved this with a clever trick: shifted window attention.
Instead of computing attention across the entire image, Swin computes it within local windows. Then, in the next layer, it shifts the windows by half a window — so information can flow between them. The result is linear complexity instead of quadratic.
┌─────────────────────────────────────────────────────────────┐
│ SWIN: SHIFTED WINDOWS │
│ │
│ Layer 1: Standard Windows (7×7 patches each) │
│ ┌─────────┬─────────┬─────────┐ │
│ │ │ │ │ │
│ │ Win 1 │ Win 2 │ Win 3 │ │
│ │ │ │ │ │
│ ├─────────┼─────────┼─────────┤ │
│ │ │ │ │ │
│ │ Win 4 │ Win 5 │ Win 6 │ │
│ │ │ │ │ │
│ ├─────────┼─────────┼─────────┤ │
│ │ │ │ │ │
│ │ Win 7 │ Win 8 │ Win 9 │ │
│ │ │ │ │ │
│ └─────────┴─────────┴─────────┘ │
│ │
│ Layer 2: Shifted Windows (by half a window) │
│ ┌────┬─────────┬─────────┬────┐ │
│ │ │ │ │ │ │
│ │ │ Win A │ Win B │ │ │
│ ├────┼─────────┼─────────┼────┤ │
│ │ │ │ │ │ │
│ │Win C│ Win D │ Win E │Win F│ │
│ ├────┼─────────┼─────────┼────┤ │
│ │ │ │ │ │ │
│ │ │ Win G │ Win H │ │ │
│ └────┴─────────┴─────────┴────┘ │
│ │
│ Patches that were in different windows now interact. │
│ Information flows across the entire image over layers. │
└─────────────────────────────────────────────────────────────┘
Swin became a general-purpose backbone. It hit 87.3% top-1 on ImageNet, beat state-of-the-art on object detection (+2.7 mAP), and crushed segmentation (+3.2 mIoU). It won ICCV 2021 Best Paper.
Why it matters: Swin made Transformers practical for real-world vision — not just classification, but detection and segmentation too.
┌─────────────────────────────────────────────────────────────┐
│ SWIN: HIERARCHICAL FEATURES │
│ │
│ Stage 1 Stage 2 Stage 3 Stage 4 │
│ H/4 × W/4 H/8 × W/8 H/16 × W/16 H/32×W/32│
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────┐ │
│ │ █ █ █ █ │ │ ██ ██ █ │ │ ████ ██ │ │█████│ │
│ │ █ █ █ █ │ │ ██ ██ █ │ │ ████ ██ │ │█████│ │
│ │ █ █ █ █ │ │ ██ ██ █ │ │ ████ ██ │ │█████│ │
│ └─────────┘ └─────────┘ └─────────┘ └─────┘ │
│ 96 channels 192 channels 384 channels 768 ch. │
│ │
│ Like CNNs, Swin builds a feature pyramid: │
│ • High resolution at stage 1 → good for small objects │
│ • Low resolution at stage 4 → good for large objects │
│ This is exactly what object detection and segmentation │
│ need. It makes Swin a drop-in replacement for ResNet. │
└─────────────────────────────────────────────────────────────┘
🧩 Paper 3: Scaling Intelligently (EfficientNet, 2020)
Before EfficientNet, scaling CNNs was an art, not a science. Want a better model? Make it deeper (ResNet). Or wider (WideResNet). Or higher-resolution. But nobody knew the optimal way to scale.
Google's EfficientNet introduced compound scaling: a single coefficient that uniformly scales depth, width, and resolution using fixed ratios.
┌─────────────────────────────────────────────────────────────┐
│ COMPOUND SCALING │
│ │
│ Traditional scaling (independent): │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Deeper │ │ Wider │ │ Higher │ │
│ │ only │ │ only │ │ Res only│ │
│ └─────────┘ └─────────┘ └─────────┘ │
│ Each gives diminishing returns. │
│ │
│ Compound scaling (together): │
│ ┌─────────────────────────────────────┐ │
│ │ Depth × Width × Resolution │ │
│ │ d = α^φ │ │
│ │ w = β^φ │ │
│ │ r = γ^φ │ │
│ │ subject to: α · β² · γ² ≈ 2 │ │
│ └─────────────────────────────────────┘ │
│ Multiplicative benefits. │
│ │
│ Intuition: Like a camera. To take the best photo, │
│ you need a bigger sensor (width), a better zoom │
│ (resolution), AND more lenses (depth) — together. │
└─────────────────────────────────────────────────────────────┘
The result was a family of models that achieved state-of-the-art accuracy with up to 10× better efficiency. EfficientNet-B0 matched ResNet-50 with 5× fewer parameters and 10× fewer FLOPs. EfficientNet-B7 hit 84.3% top-1 while being 8.4× smaller than the previous best.
┌─────────────────────────────────────────────────────────────┐
│ EFFICIENTNET: THE FAMILY │
│ │
│ Model Resolution Params FLOPs Top-1 │
│ ───────────────────────────────────────────────────── │
│ B0 224 5.3M 0.39B 77.1% │
│ B1 240 7.8M 0.70B 79.1% │
│ B2 260 9.2M 1.0B 80.1% │
│ B3 300 12M 1.8B 81.6% │
│ B4 380 19M 4.2B 82.9% │
│ B5 456 30M 9.9B 83.6% │
│ B6 528 43M 19B 84.0% │
│ B7 600 66M 37B 84.3% │
│ │
│ B0 matches ResNet-50 with 5× fewer params, 10× fewer FLOPs│
│ B7 matches GPipe (556M params) with 8.4× fewer params. │
└─────────────────────────────────────────────────────────────┘
Why it matters: EfficientNet showed how to scale optimally, not just more. It's the difference between throwing more GPUs at a problem and actually designing a better model.
🧩 Paper 4: Learning Without Labels (BEiT, 2022)
ViT needed 300 million labeled images. But labels are expensive — someone has to manually annotate every image.
Microsoft's BEiT asked: can we learn from unlabeled images, like BERT learns from unlabeled text?
The answer was yes. BEiT adapts BERT's masked language modeling to vision:
- Mask ~40% of image patches.
- Train the model to predict the visual tokens of masked patches (using a discrete VAE).
- Learn rich, semantic representations without any labels.
┌─────────────────────────────────────────────────────────────┐
│ BEiT: MASKED IMAGE MODELING │
│ │
│ Step 1: Tokenize image with discrete VAE │
│ ┌────┬────┬────┬────┐ │
│ │ T1 │ T2 │ T3 │ T4 │ ← Each patch → one visual token │
│ ├────┼────┼────┼────┤ │
│ │ T5 │ T6 │ T7 │ T8 │ │
│ ├────┼────┼────┼────┤ │
│ │ T9 │T10 │T11 │T12 │ │
│ ├────┼────┼────┼────┤ │
│ │T13 │T14 │T15 │T16 │ │
│ └────┴────┴────┴────┘ │
│ │
│ Step 2: Mask ~40% of patches │
│ ┌────┬────┬────┬────┐ │
│ │ T1 │????│ T3 │????│ ← ???? = [MASK] token │
│ ├────┼────┼────┼────┤ │
│ │????│ T6 │????│ T8 │ │
│ ├────┼────┼────┼────┤ │
│ │ T9 │????│T11 │????│ │
│ ├────┼────┼────┼────┤ │
│ │T13 │T14 │????│T16 │ │
│ └────┴────┴────┴────┘ │
│ │
│ Step 3: Predict visual tokens of masked patches │
│ ┌────┬────┬────┬────┐ │
│ │ T1 │ T2'│ T3 │ T4'│ ← T2', T4' = predictions │
│ ├────┼────┼────┼────┤ │
│ │ T5'│ T6 │ T7'│ T8 │ │
│ ├────┼────┼────┼────┤ │
│ │ T9 │ T10│T11 │ T12│ │
│ ├────┼────┼────┼────┤ │
│ │T13 │T14 │T15'│T16 │ │
│ └────┴────┴────┴────┘ │
│ │
│ Loss: cross-entropy between predicted and true tokens │
│ No labels needed. The true tokens come from the VAE. │
└─────────────────────────────────────────────────────────────┘
BEiT-L achieved 85.2% top-1 on ImageNet — outperforming supervised ViT (82.3%) — and matched models trained on 10× more data.
Why it matters: BEiT enabled learning from unlabeled data. That's how humans learn — by observing the world, not by being told the answer to everything.
┌─────────────────────────────────────────────────────────────┐
│ PREDICT TOKENS, NOT PIXELS │
│ │
│ Predicting pixels: │
│ • Forces model to learn edges, colors, textures │
│ • Low-level details, not semantics │
│ • Like predicting individual letters of a word │
│ │
│ Predicting visual tokens: │
│ • Forces model to learn what is in the patch │
│ • High-level semantics │
│ • Like predicting whole words │
│ │
│ This is why BEiT outperforms pixel-prediction methods. │
│ It's exactly the same reason BERT predicts words, │
│ not characters. │
└─────────────────────────────────────────────────────────────┘
🧩 Paper 5: Making It Deployable (TinyBERT, 2020)
BERT is powerful. BERT is also huge — 110 million parameters. Too slow for real-time apps. Too big for a phone.
TinyBERT solved this with knowledge distillation: train a small model (the student) to mimic a large model (the teacher). Two stages: general distillation on unlabeled text, then task-specific distillation.
Distillation happens at four levels: embeddings, attention matrices, hidden states, and predictions. The result: 7.5× smaller, 9.4× faster, while retaining 96.8% of BERT's performance.
┌─────────────────────────────────────────────────────────────┐
│ TINYBERT: FOUR-LEVEL DISTILLATION │
│ │
│ Teacher (BERT-Base, 110M params) │
│ ┌─────────────────────────────────────┐ │
│ │ Layer 1 │ Layer 2 │ ... │ Layer 12│ │
│ └─────────────────────────────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────┐ │
│ │ Student (TinyBERT, 14.5M params) │ │
│ │ Layer 1 │ Layer 2 │ Layer 3│L4 │ │
│ └─────────────────────────────────────┘ │
│ │
│ Distillation happens at 4 levels: │
│ 1. Embedding layer ────► match embeddings │
│ 2. Attention matrices ────► match where teacher looks │
│ 3. Hidden states ────► match internal representations│
│ 4. Prediction logits ────► match final outputs │
│ │
│ Two stages: │
│ Stage 1: General distillation (unlabeled text) │
│ Stage 2: Task-specific distillation (labeled task data) │
│ │
│ Result: 96.8% of BERT's accuracy, 7.5× smaller, 9.4× faster│
└─────────────────────────────────────────────────────────────┘
Why it matters: TinyBERT made powerful models actually deployable. It's the bridge between research and reality.
┌─────────────────────────────────────────────────────────────┐
│ TINYBERT VS DISTILBERT │
│ │
│ Model GLUE Avg Params Speedup │
│ ───────────────────────────────────────────────────── │
│ BERT-Base 79.2 110M 1× │
│ DistilBERT 77.0 66M 1.6× │
│ TinyBERT-4 76.8 14.5M 9.4× │
│ TinyBERT-6 79.0 67M 2.3× │
│ │
│ TinyBERT-4 is 4.5× smaller than DistilBERT │
│ but nearly as accurate (76.8 vs 77.0). │
│ Why? Four-level distillation transfers more knowledge │
│ than logit-only distillation. │
└─────────────────────────────────────────────────────────────┘
🔗 The Through-Line
Together, these five papers tell a story:
┌─────────────────────────────────────────────────────────────┐
│ THE EFFICIENT AI STORY ARC │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ ViT (2021) │ │
│ │ "Transformers can see" │ │
│ │ Proved the architecture is general-purpose │ │
│ └─────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Swin (2021) │ │
│ │ "Transformers can be efficient" │ │
│ │ Linear complexity + hierarchical features │ │
│ └─────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ EfficientNet (2020) │ │
│ │ "Scaling can be principled" │ │
│ │ Compound scaling: depth + width + resolution │ │
│ └─────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ BEiT (2022) │ │
│ │ "Models can learn without labels" │ │
│ │ Masked image modeling — like BERT for vision │ │
│ └─────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ TinyBERT (2020) │ │
│ │ "Models can be compressed dramatically" │ │
│ │ Four-level distillation — deployable on any device │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ Together: Smarter, not just bigger. │
│ This is the real research direction pushing AI toward │
│ near-human capability. │
└─────────────────────────────────────────────────────────────┘
This is the real research direction pushing AI toward near-human capability. Not by making models bigger — but by making them smarter, more efficient, and more general.
⚠️ The Gaps That Remain
Despite this progress, big problems remain:
┌─────────────────────────────────────────────────────────────┐
│ SEVEN RESEARCH GAPS │
│ │
│ 1. NO UNIFIED EFFICIENCY BENCHMARK │
│ Every paper uses different metrics. │
│ We can't compare fairly. │
│ → Solution: Standardized benchmark (like MLPerf) │
│ │
│ 2. MULTI-MODAL DISTILLATION MISSING │
│ TinyBERT = text; ViT/Swin = vision. │
│ No one distills CLIP or GPT-4V. │
│ → Solution: Multi-modal distillation frameworks │
│ │
│ 3. HARDWARE-AWARE EFFICIENCY │
│ FLOPs ≠ real latency. │
│ Paper-efficient ≠ phone-efficient. │
│ → Solution: Hardware-aware NAS and benchmarking │
│ │
│ 4. SELF-SUPERVISION IS EXPENSIVE │
│ BEiT is powerful but costly to pre-train. │
│ → Solution: Efficient self-supervised methods │
│ │
│ 5. NO EFFICIENT VISION-LANGUAGE MODELS │
│ CLIP and GPT-4V are huge. │
│ We need small, fast equivalents. │
│ → Solution: Efficient multi-modal architectures │
│ │
│ 6. INTERPRETABILITY IS IGNORED │
│ None of these papers address explainability. │
│ → Solution: XAI for efficient models │
│ │
│ 7. CONTINUAL LEARNING │
│ All assume static datasets. │
│ The real world changes. │
│ → Solution: Efficient continual learning │
└─────────────────────────────────────────────────────────────┘
🚀 What's Next
The next frontier is efficient, general-purpose, multi-modal AI — models that:
┌─────────────────────────────────────────────────────────────┐
│ THE NEXT FRONTIER │
│ │
│ ✅ Match human performance on any task │
│ ✅ Run on any device (phone, sensor, laptop) │
│ ✅ Learn from any data (labeled or not) │
│ ✅ Adapt continuously without retraining │
│ ✅ Explain their decisions │
│ │
│ This is the direction AI is heading. │
│ And it's not about making models bigger. │
│ It's about making them smarter. │
│ │
│ The giants will always have their place. │
│ But the future belongs to the efficient. │
└─────────────────────────────────────────────────────────────┘
💡 The Bottom Line
For years, we thought bigger meant better. But the real breakthroughs — the ones pushing AI toward human-level capability — came from a different direction.
· ViT showed Transformers could see.
· Swin made them efficient.
· EfficientNet showed how to scale.
· BEiT learned without labels.
· TinyBERT made it deployable.
Together, they define the efficient AI movement. And they prove something important:
You don't need a 175-billion-parameter model to be smart. You need a better architecture, a smarter training method, and a principled way to scale.
That's the quiet revolution. And it's just getting started.
If you found this useful, share it with someone who thinks AI is just about bigger models. And if you're building AI systems — I'd love to hear what efficiency trade-offs you're making.
📚 References
- Dosovitskiy, Alexey; Beyer, Lucas; Kolesnikov, Alexander; Weissenborn, Dirk; Zhai, Xiaohua; Unterthiner, Thomas; Dehghani, Mostafa; Minderer, Matthias; Heigold, Georg; Gelly, Sylvain; Uszkoreit, Jakob; and Houlsby, Neil (2021). An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. International Conference on Learning Representations (ICLR) 2021. arXiv identifier: 2010.11929.
- Liu, Ze; Lin, Yutong; Cao, Yue; Hu, Han; Wei, Yixuan; Zhang, Zheng; Lin, Stephen; and Guo, Baining (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. International Conference on Computer Vision (ICCV) 2021. arXiv identifier: 2103.14030.
- Tan, Mingxing and Le, Quoc V. (2020). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. International Conference on Machine Learning (ICML) 2019 / Springer. arXiv identifier: 1905.11946.
- Bao, Hangbo; Dong, Li; Piao, Songhao; and Wei, Furu (2022). BEiT: BERT Pre-Training of Image Transformers. International Conference on Learning Representations (ICLR) 2022. arXiv identifier: 2106.08254.
- Jiao, Xiaoqi; Yin, Yichun; Shang, Lifeng; Jiang, Xin; Chen, Xiao; Li, Linlin; Wang, Fang; and Liu, Qun (2020). TinyBERT: Distilling BERT for Natural Language Understanding. International Conference on Learning Representations (ICLR) 2020 / Springer. arXiv identifier: 1909.10351.
Tags: #AI #DeepLearning #MachineLearning #VisionTransformer #EfficientAI #Transformers #TechBlog #ArtificialIntelligence #SwinTransformer #EfficientNet #BEiT #TinyBERT
-
來源：DEV Community · dev.to