.NET 電腦視覺整合指南

Computer Vision with .NET

A deep-dive walkthrough of integrating computer vision into .NET applications — covering image classification (ML.NET and ONNX Runtime), object detection with YOLO-style models, OCR (Azure AI Vision, Document Intelligence, Tesseract), face and object recognition and the legal and ethical constraints around them, image embeddings and visual search, Azure AI Vision, OpenCV integration with OpenCvSharp, image processing APIs (ImageSharp, SkiaSharp and others), multimodal LLMs versus classical vision, and building a production vision service in ASP.NET Core.

Table of Contents

1. Introduction
2. The Computer Vision Landscape in .NET
3. Image Classification
4. Object Detection
5. OCR
6. Face and Object Recognition
7. Image Embeddings
8. Azure AI Vision
9. OpenCV Integration
10. Image Processing APIs
11. Multimodal LLMs vs. Classical Computer Vision
12. Building a Vision Service in ASP.NET Core
13. Evaluating Vision Systems
14. Choosing the Right Approach
15. Common Pitfalls
16. Quick Reference Table
17. Conclusion

Introduction

Computer vision is the field of getting software to understand images and video: what is in a photo, where each object is, what a document says, whether two pictures show the same thing. For .NET developers it has become practical in two ways at once: managed cloud services (Azure AI Vision and Document Intelligence) turn many tasks into a single API call, and the ONNX ecosystem lets you run state-of-the-art open models inside your own process, on your own hardware, with no Python in production.

Image / video frame
  ↓
Decode + preprocess        (ImageSharp / SkiaSharp / OpenCV: resize, normalize, orient)
  ↓
Model                      (cloud API  |  ONNX Runtime  |  ML.NET  |  multimodal LLM)
  ↓
Postprocess                (softmax, boxes + non-max suppression, text assembly, thresholds)
  ↓
Business logic             (store, search, flag for review, trigger a workflow)

This is the pipeline behind every technique in this guide. Most vision bugs hide in the first and third stages — preprocessing that doesn't match what the model was trained on, and postprocessing that misreads the model's output — not in the model itself. The guide's emphasis follows: how to wire each task correctly, how to choose between a managed API, a self-hosted model, and a multimodal LLM, and how to measure whether the result is good enough.

A note on versions and services. Cloud vision services evolve and retire features (older Computer Vision API versions, Custom Vision, and some Face capabilities have been retired or restricted over time), SDK package names and signatures change between releases, and model output formats differ across model versions. Treat the samples here as illustrating the shape of each integration, compile them against the package versions you install, and check current service documentation for availability, regions, limits, and lifecycle status before committing to a design.

1. The Computer Vision Landscape in .NET

The tasks

CLASSIFICATION      What is in this image?                      "cat", "defective", "invoice"      one label (or a few) per image
OBJECT DETECTION    What objects are where?                     boxes + labels + confidence         "3 cars at these coordinates"
SEGMENTATION        Which pixels belong to which object?        per-pixel masks                     medical imaging, background removal
OCR                 What text is in this image?                 text + positions                    receipts, IDs, signs, scanned PDFs
DOCUMENT ANALYSIS   Structure and fields of a document          tables, key-value pairs             invoices, forms
FACE TASKS          Detect, verify, identify faces              boxes, embeddings, match scores     (heavily regulated — Section 5)
EMBEDDINGS          A vector capturing the image's meaning      float[ ]                            visual search, dedupe, recommendations
CAPTIONING / VQA    Describe or answer questions about image    natural language                    accessibility, search, assistants

The four ways to get a vision capability

1. MANAGED CLOUD API       Azure AI Vision (Image Analysis, Read OCR), Azure AI Document Intelligence, Azure AI Face.
                           + No ML expertise, fast to build, maintained and improved for you.
                           - Per-call cost, data leaves your boundary (unless containers), limited customization,
                             vendor features change and retire.

2. PRE-TRAINED MODEL ON    Download an ONNX model (ResNet, MobileNet, YOLO, CLIP, ArcFace...) and run it in-process
   ONNX RUNTIME            with Microsoft.ML.OnnxRuntime.
                           + Full control, no per-call fee, data stays local, runs offline/at the edge, can use GPU.
                           - You own preprocessing/postprocessing, model hosting, performance, and licensing checks.

3. TRAIN YOUR OWN          Fine-tune a model on your labeled images — ML.NET transfer learning for classification, or
                           train in PyTorch/TensorFlow/Azure ML and export to ONNX.
                           + Best accuracy on your specific domain (defect types, your products).
                           - Needs labeled data, evaluation discipline, and MLOps.

4. MULTIMODAL LLM          Send the image to a vision-capable chat model (see Section 10).
                           + Extremely flexible, zero-shot, handles messy real-world images and open-ended questions.
                           - Cost, latency, non-determinism, hallucination risk, weaker at precise localization and
                             counting, and data-privacy considerations.

The core .NET packages

Microsoft.ML.OnnxRuntime (+ .Gpu / .DirectML)   Run ONNX models in-process (CPU, CUDA, DirectML)
Microsoft.ML + Microsoft.ML.Vision + Microsoft.ML.ImageAnalytics   ML.NET image classification via transfer learning
SixLabors.ImageSharp (+ .Drawing)               Managed, cross-platform image processing
SkiaSharp                                        Fast 2D graphics and image processing (Skia)
OpenCvSharp4 (+ runtime packages)               OpenCV bindings: classic CV, video, DNN
Azure.AI.Vision.ImageAnalysis                   Azure AI Vision Image Analysis 4.0 (captions, tags, objects, read OCR, people)
Azure.AI.DocumentIntelligence                   Document OCR, layout, and prebuilt/custom extractors
Tesseract                                        Open-source OCR engine wrapper
Microsoft.Extensions.AI                          Vision-capable chat models via IChatClient (Section 10)
System.Numerics.Tensors                          Vector math (cosine similarity) for embeddings

A decision shortcut

"Read the text/fields from documents"                 -> Azure AI Document Intelligence (or Vision Read for plain OCR)
"Describe / tag / caption general photos"             -> Azure AI Vision Image Analysis, or a multimodal LLM
"Detect common objects (people, cars, ...)"           -> Azure AI Vision Objects/People, or a YOLO ONNX model
"Classify MY specific categories (defects, SKUs)"     -> ML.NET transfer learning or a fine-tuned model exported to ONNX
"Find similar images / search images by text"         -> image embeddings (CLIP-style) + vector search
"Identify people by face"                             -> stop and read Section 5 first (legal, ethical, and access constraints)
"Process / resize / crop / draw on images"            -> ImageSharp, SkiaSharp, or OpenCvSharp

2. Image Classification

Assign a label (or probabilities over labels) to a whole image

Classification answers "what is this a picture of?" The output is a probability for each known class; you take the top one (or top-k) and apply a confidence threshold. This series' AI/ML Fundamentals guide covers the underlying concepts (training vs. inference, overfitting, precision/recall) that apply here unchanged.

Option A: Run a pre-trained model with ONNX Runtime

Models trained on large general datasets (such as ImageNet's 1,000 classes) are available in ONNX format. Running one means loading the model, preprocessing the image exactly as the model expects, running inference, and decoding the output.

// dotnet add package Microsoft.ML.OnnxRuntime
// dotnet add package SixLabors.ImageSharp
using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;

public sealed class ImageClassifier : IDisposable
{
    private readonly InferenceSession _session;                  // create ONCE and reuse — loading a model is expensive
    private readonly string _inputName;
    private readonly string[] _labels;

    // ImageNet normalization used by many torchvision-trained models — MUST match YOUR model's training
    private static readonly float[] Mean = [0.485f, 0.456f, 0.406f];
    private static readonly float[] Std  = [0.229f, 0.224f, 0.225f];

    public ImageClassifier(string modelPath, string labelsPath)
    {
        _session = new InferenceSession(modelPath);
        _inputName = _session.InputMetadata.Keys.First();
        _labels = File.ReadAllLines(labelsPath);
    }

    public IReadOnlyList<(string Label, float Probability)> Classify(Image<Rgb24> image, int topK = 3)
    {
        // 1. PREPROCESS: resize + center-crop to 224x224, scale to 0..1, normalize, order as NCHW (batch, channel, height, width)
        using var resized = image.Clone(x => x.Resize(new ResizeOptions { Size = new Size(224, 224), Mode = ResizeMode.Crop }));

        var tensor = new DenseTensor<float>([1, 3, 224, 224]);
        resized.ProcessPixelRows(accessor =>
        {
            for (int y = 0; y < accessor.Height; y++)
            {
                Span<Rgb24> row = accessor.GetRowSpan(y);
                for (int x = 0; x < row.Length; x++)
                {
                    tensor[0, 0, y, x] = (row[x].R / 255f - Mean[0]) / Std[0];
                    tensor[0, 1, y, x] = (row[x].G / 255f - Mean[1]) / Std[1];
                    tensor[0, 2, y, x] = (row[x].B / 255f - Mean[2]) / Std[2];
                }
            }
        });

        // 2. INFERENCE
        using var results = _session.Run([NamedOnnxValue.CreateFromTensor(_inputName, tensor)]);
        float[] logits = results.First().AsEnumerable<float>().ToArray();

        // 3. POSTPROCESS: softmax turns raw scores (logits) into probabilities that sum to 1
        float max = logits.Max();
        float[] exp = logits.Select(l => MathF.Exp(l - max)).ToArray();     // subtract max for numerical stability
        float sum = exp.Sum();

        return exp.Select((e, i) => (Label: _labels[i], Probability: e / sum))
                  .OrderByDescending(x => x.Probability)
                  .Take(topK)
                  .ToList();
    }

    public void Dispose() => _session.Dispose();
}

THE PREPROCESSING MUST MATCH TRAINING. Check your model's documentation for:
  - INPUT SIZE        (224x224, 256x256, 384x384 ...)
  - CHANNEL ORDER     RGB vs BGR (OpenCV loads BGR; most deep-learning models expect RGB)
  - LAYOUT            NCHW (batch, channels, height, width) vs NHWC
  - SCALING           0..1 vs 0..255 vs -1..1
  - NORMALIZATION     per-channel mean/std (or none)
  - RESIZE POLICY     stretch vs. preserve aspect ratio with crop or padding
A mismatch doesn't throw an error — it just silently degrades accuracy. This is the #1 cause of "the model works in Python
but gives garbage in .NET."

Option B: Train your own classifier with ML.NET (transfer learning)

For your categories — product types, defect vs. OK, plant diseases — ML.NET can fine-tune the top of a pre-trained network on your labeled images. This is transfer learning: the pre-trained network already knows general visual features, so you need far less data (often hundreds of images per class rather than millions).

// dotnet add package Microsoft.ML
// dotnet add package Microsoft.ML.Vision
// dotnet add package Microsoft.ML.ImageAnalytics
// dotnet add package SciSharp.TensorFlow.Redist        (the TensorFlow runtime ML.NET's vision trainer uses)
using Microsoft.ML;
using Microsoft.ML.Vision;

public class ImageData
{
    public string ImagePath { get; set; } = "";
    public string Label { get; set; } = "";              // e.g. the name of the folder the image sits in
}

public class ImagePrediction
{
    public string PredictedLabel { get; set; } = "";
    public float[] Score { get; set; } = [];
}

var mlContext = new MLContext(seed: 1);

// Folder layout:  images/<label>/<file>.jpg   — one subfolder per class
IEnumerable<ImageData> images = Directory.EnumerateFiles("images", "*.jpg", SearchOption.AllDirectories)
    .Select(path => new ImageData { ImagePath = path, Label = Directory.GetParent(path)!.Name });

IDataView data = mlContext.Data.ShuffleRows(mlContext.Data.LoadFromEnumerable(images));

// Convert the string label to a numeric "key" and load the raw image bytes
var preprocess = mlContext.Transforms.Conversion.MapValueToKey("LabelAsKey", nameof(ImageData.Label))
    .Append(mlContext.Transforms.LoadRawImageBytes("Image", null, nameof(ImageData.ImagePath)));

IDataView processed = preprocess.Fit(data).Transform(data);

var split = mlContext.Data.TrainTestSplit(processed, testFraction: 0.3, seed: 1);
var validation = mlContext.Data.TrainTestSplit(split.TestSet, testFraction: 0.5, seed: 1);
IDataView trainSet = split.TrainSet, validationSet = validation.TrainSet, testSet = validation.TestSet;

var options = new ImageClassificationTrainer.Options
{
    FeatureColumnName = "Image",
    LabelColumnName = "LabelAsKey",
    ValidationSet = validationSet,
    Arch = ImageClassificationTrainer.Architecture.ResnetV2101,      // the pre-trained network to build on
    MetricsCallback = Console.WriteLine,                             // progress and validation accuracy per epoch
    TestOnTrainSet = false,
    ReuseTrainSetBottleneckCachedValues = true,                      // cache the expensive feature extraction between epochs
    ReuseValidationSetBottleneckCachedValues = true,
};

var pipeline = mlContext.MulticlassClassification.Trainers.ImageClassification(options)
    .Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel"));

ITransformer model = pipeline.Fit(trainSet);

// Evaluate on images the model never trained on (see AI/ML Fundamentals, Section 8)
var predictions = model.Transform(testSet);
var metrics = mlContext.MulticlassClassification.Evaluate(predictions, labelColumnName: "LabelAsKey", predictedLabelColumnName: "PredictedLabel");
Console.WriteLine($"Micro-accuracy: {metrics.MicroAccuracy:P1}   Macro-accuracy: {metrics.MacroAccuracy:P1}");

mlContext.Model.Save(model, trainSet.Schema, "imageClassifier.zip");

(This follows the pattern of ML.NET's image-classification tutorials; the trainer's architecture options and required runtime packages change over time, so check the current ML.NET documentation. For deeper modeling or newer architectures, a common route is to train in PyTorch and export to ONNX, then run with ONNX Runtime as in Option A.)

Option C: Managed tagging

Azure AI Vision's Image Analysis returns general tags and captions for a photo with no training at all (Section 7) — a fine fit when "is there a dog in it?" level labeling suffices and your categories are common ones.

Classification practicalities

CONFIDENCE THRESHOLDS     A classifier always returns SOME label, even for an image unlike anything it has seen. Apply a threshold
                          and add an "unknown / needs human review" path — don't act on a 41% prediction as if it were certain.
CLASS IMBALANCE           If 95% of images are "OK," accuracy is misleading (see AI/ML Fundamentals, Section 9). Track precision, recall,
                          and per-class results, and use a confusion matrix.
DATA QUALITY OVER MODEL   Consistent labels and images representative of PRODUCTION conditions (lighting, angle, camera, backgrounds) matter
                          more than the architecture. Models trained on clean studio photos fail on blurry phone snapshots.
SPLIT BY SOURCE           Near-duplicate images (frames of the same video, photos of the same object) must stay on one side of the
                          train/test split, or you'll measure memorization.
AUGMENTATION              Flips, crops, brightness/contrast changes, and small rotations expand your data and improve robustness.
MULTI-LABEL               If an image can have several labels at once, you need a multi-label setup (independent probabilities), not softmax.

3. Object Detection

Find objects and their locations

Classification says what; object detection says what and where: a list of bounding boxes, each with a label and a confidence score. It powers counting, safety monitoring, inventory, cropping, and any workflow that needs to act on a specific region.

Output of a detector for one image:
  [ { label: "person", confidence: 0.91, box: (x=120, y=64, w=88, h=210) },
    { label: "car",    confidence: 0.84, box: (x=410, y=190, w=260, h=130) }, ... ]

Running a YOLO-style model with ONNX Runtime

YOLO-family models are the most common real-time detectors, and exported ONNX versions run well from .NET. The wiring has three parts that must all be right: letterbox preprocessing, inference, and postprocessing (decode boxes, filter by confidence, and apply non-maximum suppression).

using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;
using RectangleF = System.Drawing.RectangleF;       // avoid ambiguity with ImageSharp's own RectangleF

public record Detection(string Label, float Confidence, RectangleF Box);

public sealed class ObjectDetector : IDisposable
{
    private const int InputSize = 640;
    private readonly InferenceSession _session;
    private readonly string _inputName;
    private readonly string[] _labels;

    public ObjectDetector(string modelPath, string[] labels)
    {
        _session = new InferenceSession(modelPath);
        _inputName = _session.InputMetadata.Keys.First();
        _labels = labels;
    }

    public IReadOnlyList<Detection> Detect(Image<Rgb24> original, float confThreshold = 0.25f, float iouThreshold = 0.45f)
    {
        // 1. LETTERBOX: scale to fit 640x640 while PRESERVING aspect ratio, padding the rest with gray
        float scale = Math.Min((float)InputSize / original.Width, (float)InputSize / original.Height);
        int newW = (int)(original.Width * scale), newH = (int)(original.Height * scale);
        int padX = (InputSize - newW) / 2, padY = (InputSize - newH) / 2;

        using var resized = original.Clone(x => x.Resize(newW, newH));

        var input = new DenseTensor<float>([1, 3, InputSize, InputSize]);
        input.Fill(114f / 255f);                                           // padding color
        resized.ProcessPixelRows(accessor =>
        {
            for (int y = 0; y < accessor.Height; y++)
            {
                Span<Rgb24> row = accessor.GetRowSpan(y);
                for (int x = 0; x < row.Length; x++)
                {
                    input[0, 0, y + padY, x + padX] = row[x].R / 255f;     // YOLO-style models typically want RGB scaled 0..1
                    input[0, 1, y + padY, x + padX] = row[x].G / 255f;
                    input[0, 2, y + padY, x + padX] = row[x].B / 255f;
                }
            }
        });

        // 2. INFERENCE
        using var results = _session.Run([NamedOnnxValue.CreateFromTensor(_inputName, input)]);
        Tensor<float> output = results.First().AsTensor<float>();         // YOLOv8-style layout: [1, 4 + numClasses, numAnchors]

        int numClasses = output.Dimensions[1] - 4;
        int numAnchors = output.Dimensions[2];

        // 3. POSTPROCESS: decode candidate boxes
        var candidates = new List<Detection>();
        for (int i = 0; i < numAnchors; i++)
        {
            int bestClass = 0; float bestScore = 0;
            for (int c = 0; c < numClasses; c++)
            {
                float score = output[0, 4 + c, i];
                if (score > bestScore) { bestScore = score; bestClass = c; }
            }
            if (bestScore < confThreshold) continue;

            float cx = output[0, 0, i], cy = output[0, 1, i], w = output[0, 2, i], h = output[0, 3, i];

            // Undo the letterbox: model coordinates -> original image coordinates
            float x1 = (cx - w / 2 - padX) / scale;
            float y1 = (cy - h / 2 - padY) / scale;
            candidates.Add(new Detection(_labels[bestClass], bestScore, new RectangleF(x1, y1, w / scale, h / scale)));
        }

        // 4. NON-MAXIMUM SUPPRESSION: remove duplicate overlapping boxes for the same object
        return NonMaxSuppression(candidates, iouThreshold);
    }

    private static List<Detection> NonMaxSuppression(List<Detection> detections, float iouThreshold)
    {
        var kept = new List<Detection>();

        foreach (var group in detections.GroupBy(d => d.Label))           // per-class NMS
        {
            var remaining = group.OrderByDescending(d => d.Confidence).ToList();
            while (remaining.Count > 0)
            {
                Detection best = remaining[0];
                kept.Add(best);
                remaining.RemoveAt(0);
                remaining.RemoveAll(d => IoU(best.Box, d.Box) > iouThreshold);
            }
        }
        return kept;
    }

    private static float IoU(RectangleF a, RectangleF b)
    {
        RectangleF inter = RectangleF.Intersect(a, b);
        float interArea = inter.IsEmpty ? 0 : inter.Width * inter.Height;
        float union = a.Width * a.Height + b.Width * b.Height - interArea;
        return union <= 0 ? 0 : interArea / union;
    }

    public void Dispose() => _session.Dispose();
}

The concepts behind that code

BOUNDING BOX        A rectangle: (x, y, width, height) or (x1, y1, x2, y2). Models may output center-based (cx, cy, w, h) —
                    convert carefully.
CONFIDENCE          The model's score for the detection; filter by a threshold. Lower = more detections and more false
                    positives; higher = fewer false positives but more misses.
IoU                 "Intersection over Union": the overlap between two boxes divided by their combined area (0 = no overlap,
                    1 = identical). Used both for NMS and for evaluating detections against ground truth.
NMS                 Detectors emit many overlapping boxes per object. NMS keeps the most confident box and discards others
                    that overlap it above the IoU threshold.
LETTERBOXING        Preserving aspect ratio (padding) instead of stretching — stretched images distort object shapes and hurt
                    accuracy, and you MUST map boxes back to the original image's coordinates afterward.

Model-specific details to verify

OUTPUT LAYOUT   Differs by model family and export settings: some outputs include a separate "objectness" score, some are transposed
                ([1, numAnchors, 4 + classes]), and some exports bake NMS into the model. Inspect the output shape
                (session.OutputMetadata) and the model's documentation — the code above matches the YOLOv8-style layout.
LABELS          The class-name list must match the model's training (80 COCO classes for common pretrained checkpoints).
LICENSE         Check the license of the model and its weights before commercial use — some popular detectors use licenses with
                obligations (such as copyleft terms) that matter for products.
PERFORMANCE     Indexing a Tensor per element is convenient but slow; for throughput, copy to a float[] and index with arithmetic.
                Use GPU execution providers (CUDA, DirectML) and smaller or quantized (INT8) models where latency matters.

Managed and trainable alternatives

AZURE AI VISION    The Image Analysis API returns detected objects, people, and tags with bounding boxes — no model to host,
                   good for common categories (Section 7).
CUSTOM DETECTORS   For YOUR objects (a specific part, a defect type), fine-tune a detector on labeled images (label with a tool like
                   CVAT or Label Studio), train in PyTorch or Azure ML, export to ONNX, and run with the code above.
ML.NET             Offers object detection via Model Builder and Azure-based training workflows; check current documentation for
                   what it supports.

Using detections

COUNTING            Count boxes per class — but beware occlusion, tiny objects, and duplicates; validate counts against ground truth.
CROPPING            Crop each box and send it to a second model (a classifier, OCR, or a face model) — a common multi-stage pipeline.
REGION RULES        "Alert when a person is inside this polygon" — geometry on top of detections.
TRACKING (VIDEO)    Detection works per frame. To follow an object across frames you need a tracking layer (e.g. SORT/ByteTrack-style
                    algorithms) that associates detections over time.
THRESHOLDS PER CLASS  Tune confidence per class — a safety-critical class may deserve a lower threshold (accepting false alarms)
                    than a convenience feature.

4. OCR

Turning pixels of text into text

Optical Character Recognition extracts text from images — scanned documents, photos of receipts and signs, screenshots, ID cards. Modern OCR returns not just the text but structure: lines and words with bounding boxes, reading order, and often tables and fields.

The OCR options

AZURE AI VISION READ        Cloud OCR for printed and handwritten text in photos and documents, many languages. Part of the
                            Image Analysis 4.0 API. Good for general "get me the text" needs.

AZURE AI DOCUMENT           Document-focused: layout (paragraphs, tables, selection marks), prebuilt models (invoices, receipts,
INTELLIGENCE                IDs, business cards, tax forms) that return structured FIELDS, and custom models trained on your
                            forms. The right tool for documents rather than photos.

TESSERACT                   Open-source engine, runs locally. Free and offline, but accuracy depends heavily on image quality and
                            preprocessing; weaker on photos, handwriting, and complex layouts.

OTHER LOCAL ENGINES         PaddleOCR (via .NET wrappers) and platform OCR (such as Windows' built-in OCR) are options with
                            different accuracy, language, and portability trade-offs.

MULTIMODAL LLMs             Vision-capable chat models can read text and extract structured fields from messy images
                            (Section 10) — flexible, but slower and costlier, and they can hallucinate text.

Azure AI Vision: Read

// dotnet add package Azure.AI.Vision.ImageAnalysis
using Azure;
using Azure.AI.Vision.ImageAnalysis;

var client = new ImageAnalysisClient(new Uri(endpoint), new AzureKeyCredential(key));     // or an Entra ID TokenCredential

ImageAnalysisResult result = await client.AnalyzeAsync(
    BinaryData.FromBytes(imageBytes),               // or pass a Uri to an image the service can fetch
    VisualFeatures.Read);

foreach (DetectedTextBlock block in result.Read.Blocks)
    foreach (DetectedTextLine line in block.Lines)
    {
        Console.WriteLine(line.Text);

        foreach (DetectedTextWord word in line.Words)
            Console.WriteLine($"   {word.Text}  (confidence {word.Confidence:P0})");
    }

Azure AI Document Intelligence: layout and prebuilt fields

// dotnet add package Azure.AI.DocumentIntelligence
using Azure;
using Azure.AI.DocumentIntelligence;

var docClient = new DocumentIntelligenceClient(new Uri(endpoint), new AzureKeyCredential(key));

// "prebuilt-layout" returns text, tables, and structure; "prebuilt-invoice", "prebuilt-receipt", "prebuilt-idDocument"... return typed fields
Operation<AnalyzeResult> operation = await docClient.AnalyzeDocumentAsync(
    WaitUntil.Completed, "prebuilt-invoice", BinaryData.FromBytes(pdfOrImageBytes));

AnalyzeResult analysis = operation.Value;

foreach (AnalyzedDocument doc in analysis.Documents)
{
    if (doc.Fields.TryGetValue("VendorName", out DocumentField? vendor))
        Console.WriteLine($"Vendor: {vendor.ValueString}  (confidence {vendor.Confidence:P0})");

    if (doc.Fields.TryGetValue("InvoiceTotal", out DocumentField? total))
        Console.WriteLine($"Total: {total.ValueCurrency?.Amount}");
}

(SDK method shapes for Document Intelligence have changed between releases; check the version you install.) Prebuilt models save enormous effort on common document types: instead of OCR-then-regex, you get named, typed fields with per-field confidence scores — the key to deciding which extractions need human review.

Tesseract: local OCR

// dotnet add package Tesseract     (also download the trained-data files, e.g. eng.traineddata, into ./tessdata)
using Tesseract;

using var engine = new TesseractEngine(@"./tessdata", "eng", EngineMode.Default);
using var pix = Pix.LoadFromFile("scan.png");
using var page = engine.Process(pix);

Console.WriteLine(page.GetText());
Console.WriteLine($"Mean confidence: {page.GetMeanConfidence():P0}");

OCR quality depends on the image — preprocess

RESOLUTION     Aim for roughly 300 DPI equivalent for documents; upscale tiny text before OCR.
CONTRAST       Convert to grayscale and apply adaptive thresholding (binarization) to separate text from uneven backgrounds.
DESKEW         Rotate skewed scans so lines are horizontal. Correct orientation (EXIF rotation, upside-down pages).
DENOISE        Remove speckle and scanner noise; avoid blurring text itself.
CROP           Remove borders, shadows, and non-text regions.
LIGHTING       Phone photos often have shadows and glare — normalize illumination.

// Typical OpenCvSharp cleanup before OCR (Section 8)
using OpenCvSharp;

using var src  = Cv2.ImRead("receipt.jpg", ImreadModes.Color);
using var gray = new Mat();
Cv2.CvtColor(src, gray, ColorConversionCodes.BGR2GRAY);

using var denoised = new Mat();
Cv2.GaussianBlur(gray, denoised, new Size(3, 3), 0);

using var binary = new Mat();
Cv2.AdaptiveThreshold(denoised, binary, 255, AdaptiveThresholdTypes.GaussianC, ThresholdTypes.Binary, blockSize: 31, c: 15);

Cv2.ImWrite("receipt-clean.png", binary);

Cloud services such as Read and Document Intelligence do much of this internally and are usually more forgiving than Tesseract; for local OCR, preprocessing is the difference between mediocre and good results.

Working with OCR output

CONFIDENCE         Use word/line/field confidence to route low-confidence results to human review rather than trusting them blindly.
VALIDATE           Cross-check extracted values: totals equal the sum of line items; dates parse; account numbers pass checksums;
                   VAT numbers match a pattern. Validation catches OCR errors that confidence scores miss.
STRUCTURE          For tables and forms use a layout-aware service rather than reconstructing columns from raw text.
LANGUAGES / SCRIPTS  Verify support for your languages and scripts, including right-to-left text and handwriting.
RAG PIPELINES      OCR is the first stage of ingesting scanned PDFs for retrieval (see this series' RAG guide, Section 3): detect low-text PDF
                   pages and route them through OCR.
SENSITIVE DATA     IDs, medical forms, and financial documents are sensitive. Control where images are sent, how long they're retained, and who
                   can see the extracted text (see the security notes in Section 11).

Measuring OCR

Evaluate with character error rate (CER) and word error rate (WER) — the edit distance between OCR output and the ground-truth text, divided by the reference length — plus field-level accuracy for structured extraction:

static double CharacterErrorRate(string reference, string hypothesis)
{
    if (reference.Length == 0) return hypothesis.Length == 0 ? 0 : 1;
    return (double)Levenshtein(reference, hypothesis) / reference.Length;
}

static int Levenshtein(string a, string b)
{
    var previous = Enumerable.Range(0, b.Length + 1).ToArray();
    for (int i = 1; i <= a.Length; i++)
    {
        var current = new int[b.Length + 1];
        current[0] = i;
        for (int j = 1; j <= b.Length; j++)
            current[j] = Math.Min(Math.Min(current[j - 1] + 1, previous[j] + 1), previous[j - 1] + (a[i - 1] == b[j - 1] ? 0 : 1));
        previous = current;
    }
    return previous[b.Length];
}

5. Face and Object Recognition

Be precise about what "face recognition" means

FACE DETECTION      "Is there a face, and where?" — boxes (and optionally landmarks such as eyes and nose). The least sensitive
                    task; used for auto-cropping, blurring, camera focus, and counting.

FACE VERIFICATION   "Are these two images the same person?" (1:1). Used for unlocking a device or matching a selfie to an ID photo.

FACE IDENTIFICATION "Who is this, among a set of known people?" (1:N). Searches a gallery of enrolled faces. The most sensitive task.

FACE ATTRIBUTES     Inferring age, gender, emotion, or other traits from a face. Scientifically contested and discontinued or
                    restricted by major providers (below). Avoid.

Treat face recognition as a legal and ethical decision first

Faces are biometric data. Using them to identify people carries obligations and, in some jurisdictions and contexts, prohibitions. This is not a footnote:

LAW AND REGULATION     Biometric identification is treated as sensitive or special-category data under laws such as the GDPR, and some
                       jurisdictions have dedicated biometric statutes (for example Illinois' BIPA, with consent, retention, and
                       private-right-of-action provisions). The EU AI Act restricts or prohibits certain biometric uses — for example
                       real-time remote biometric identification in public spaces by law enforcement (with narrow exceptions), untargeted
                       scraping of facial images to build databases, and emotion recognition in workplaces and schools. Rules differ by
                       country and are evolving: consult legal counsel before building anything that identifies people by face.

CONSENT AND PURPOSE    Obtain informed consent where required, use faces only for the stated purpose, and don't repurpose them.

DATA PROTECTION        Store face embeddings/templates as sensitive data: encryption, strict access control, minimal retention,
                       deletion on request, and a clear policy for what happens when a person withdraws consent.

BIAS AND ACCURACY      Face recognition accuracy has been shown to vary across demographic groups and imaging conditions. A system that
                       works well on average can fail badly — and unfairly — on specific groups. Evaluate by subgroup, set conservative
                       thresholds, and never make high-stakes decisions on a face match alone.

PLATFORM POLICIES      Microsoft's Azure Face service applies a Limited Access policy: identification and verification features require an
                       approved application and use case, and the earlier capabilities for inferring attributes like emotion, gender, and age
                       were retired. Other vendors have their own restrictions. Read the current terms.

ALTERNATIVES           Often a less invasive design meets the need: a badge or PIN plus a face DETECTION step for privacy blurring, 1:1
                       verification against a document the user provides, or on-device matching where templates never leave the user's device.

The technical pipeline (for lawful, consented uses)

1. DETECT    Find faces in the image (a face detector returns boxes and landmarks).
2. ALIGN     Crop and rotate so eyes and mouth sit in standard positions — recognition models expect aligned faces.
3. EMBED     A face-recognition network (ArcFace/FaceNet-style, available as ONNX models) converts each aligned face to a vector
             (commonly 128-512 numbers) such that the same person's photos land close together.
4. COMPARE   Verification: cosine similarity (or distance) between two embeddings vs. a threshold.
             Identification: nearest-neighbor search over a gallery of enrolled embeddings (vector search — see this series'
             Embeddings & Vector Search guide), then apply the threshold to the best match.
5. DECIDE    A threshold trades false accepts against false rejects (below). Add liveness detection to defeat photos and replays.

// Verification with embeddings from a face-recognition ONNX model (embedding extraction code is analogous to the classifier above)
float[] a = faceEmbedder.Embed(alignedFaceA);
float[] b = faceEmbedder.Embed(alignedFaceB);

float similarity = TensorPrimitives.CosineSimilarity(a, b);
bool samePerson = similarity >= threshold;       // choose the threshold from a labeled validation set, NOT from a blog post

FALSE ACCEPT RATE (FAR)   Different people judged to be the same — the SECURITY failure.
FALSE REJECT RATE (FRR)   The same person judged different — the USABILITY failure.
The threshold trades one for the other. Pick it on a labeled validation set representative of your users and conditions, to meet a
target FAR for your risk level, and re-measure by subgroup.

LIVENESS / ANTI-SPOOFING  Without it, a printed photo or a screen replay can fool recognition. Use liveness checks for any
                          authentication use.

Face detection in .NET (the low-risk task)

// Classic detector via OpenCvSharp — simple and dependency-light, but less accurate than modern deep-learning detectors
using OpenCvSharp;

using var cascade = new CascadeClassifier("haarcascade_frontalface_default.xml");
using var image = Cv2.ImRead("group.jpg");
using var gray = new Mat();
Cv2.CvtColor(image, gray, ColorConversionCodes.BGR2GRAY);

Rect[] faces = cascade.DetectMultiScale(gray, scaleFactor: 1.1, minNeighbors: 5, flags: HaarDetectionTypes.ScaleImage, minSize: new Size(40, 40));

foreach (Rect face in faces)
    Cv2.Rectangle(image, face, Scalar.Red, 2);        // or: blur the region to anonymize people in an image

Modern deep-learning face detectors (such as the YuNet model supported by OpenCV's DNN module, or ONNX-exported detectors) are considerably more accurate than Haar cascades, especially for non-frontal and small faces; Azure AI Vision's people/face detection and the Azure Face service are managed options for detection.

"Object recognition": specific things, not just categories

CATEGORY RECOGNITION    "That is a laptop" — classification/detection (Sections 2-3).
INSTANCE RECOGNITION    "That is THIS exact product (SKU 4471)" or "this is a photo of the Eiffel Tower" — distinguishing specific instances.
                        The modern approach is EMBEDDING-BASED: embed a reference image of each known item, embed the query image, and
                        find the nearest reference via vector search (Section 6). Adding a new product means adding an embedding — no retraining.

6. Image Embeddings

A vector that captures what an image means

An image embedding is a list of numbers (often hundreds to a thousand-plus) produced by a vision model such that visually or semantically similar images get nearby vectors — the same idea as text embeddings (see this series' Embeddings & Vector Search guide), applied to pixels. With embeddings you can compare, search, and cluster images without labeling anything.

Multimodal embeddings: images and text in one space

Models in the CLIP family are trained on image–text pairs so that an image and its description land close together in the same vector space. That unlocks cross-modal search: embed a text query ("red running shoes on a trail") and find the nearest images.

IMAGE → vector     image embedding model
TEXT  → vector     text embedding model (the PAIRED model of the SAME family)
Compare with cosine similarity: text-to-image search, image-to-image search, zero-shot classification.

Where embeddings come from

AZURE AI VISION MULTIMODAL EMBEDDINGS   A managed API that embeds images AND text into a shared space (REST endpoints for vectorizing images and text).
                                        No model to host.

CLIP-STYLE MODELS VIA ONNX              Open CLIP-family models exported to ONNX run locally with ONNX Runtime — an image encoder and a text
                                        encoder (the text side needs the matching tokenizer, which is the fiddly part).

VISION BACKBONES                        Features from image models such as ResNet or DINO-style networks work well for image-to-image similarity
                                        (not text search) — export the model so its feature (pre-classification) layer is exposed.

MULTIMODAL LLM PROVIDERS                Some providers offer multimodal embedding endpoints; check availability.

A small abstraction

public interface IImageEmbedder
{
    Task<float[]> EmbedImageAsync(byte[] imageBytes, CancellationToken ct = default);
    Task<float[]> EmbedTextAsync(string text, CancellationToken ct = default);       // only meaningful for multimodal (CLIP-style) models
}

A managed implementation is a thin HTTP call (the exact URL, API version, and request shape for Azure AI Vision's multimodal embeddings should be taken from the current documentation):

public sealed class AzureVisionEmbedder(HttpClient http, string endpoint, string key) : IImageEmbedder
{
    // Illustrative: confirm the route, api-version, and model-version in the current Azure AI Vision documentation
    public async Task<float[]> EmbedImageAsync(byte[] imageBytes, CancellationToken ct = default)
    {
        using var request = new HttpRequestMessage(HttpMethod.Post,
            $"{endpoint}/computervision/retrieval:vectorizeImage?api-version=2024-02-01&model-version=2023-04-15")
        {
            Content = new ByteArrayContent(imageBytes) { Headers = { ContentType = new("application/octet-stream") } }
        };
        request.Headers.Add("Ocp-Apim-Subscription-Key", key);

        using var response = await http.SendAsync(request, ct);
        response.EnsureSuccessStatusCode();

        var json = await response.Content.ReadFromJsonAsync<EmbeddingResponse>(cancellationToken: ct);
        return json!.Vector;
    }

    public async Task<float[]> EmbedTextAsync(string text, CancellationToken ct = default) { /* the vectorizeText counterpart */ throw new NotImplementedException(); }

    private record EmbeddingResponse(float[] Vector);
}

What you can build

// 1. INDEX: embed every catalog image once and store the vector (with the image id and metadata) in a vector store
foreach (var product in catalog)
    await store.UpsertAsync(product.Id, await embedder.EmbedImageAsync(product.ImageBytes), product.Metadata);

// 2. SEARCH BY IMAGE: "find products that look like this photo"
float[] queryVector = await embedder.EmbedImageAsync(uploadedPhotoBytes);
var similar = await store.SearchAsync(queryVector, top: 10, filter: tenantFilter);

// 3. SEARCH BY TEXT (multimodal models only): "red leather handbag"
float[] textVector = await embedder.EmbedTextAsync("red leather handbag");
var matches = await store.SearchAsync(textVector, top: 10, filter: tenantFilter);

// 4. ZERO-SHOT CLASSIFICATION: compare the image to text prompts for each candidate label — no training data
string[] labels = ["cat", "dog", "bird"];
var labelVectors = await Task.WhenAll(labels.Select(l => embedder.EmbedTextAsync($"a photo of a {l}")));
float[] imageVector = await embedder.EmbedImageAsync(imageBytes);
int best = Enumerable.Range(0, labels.Length).MaxBy(i => TensorPrimitives.CosineSimilarity(imageVector, labelVectors[i]));

USE CASES
  Visual search ("shop the look"), reverse image search          Duplicate and near-duplicate detection (moderation, DAM systems)
  Recommendations ("more like this")                              Instance recognition (match a photo to a known product)
  Clustering/organizing photo collections                         Zero-shot classification and tagging
  Retrieval for multimodal RAG (find relevant diagrams or figures)

Rules (the same as for text embeddings)

SAME MODEL for the index and the queries — different models' vectors are not comparable. Record the model name/version with stored vectors; changing
  models means re-embedding everything.
NORMALIZE / USE COSINE per the model's guidance; choose the vector index metric to match.
CACHE embeddings by content hash — re-embedding unchanged images is pure waste.
FILTER BY METADATA and permissions INSIDE the vector query (tenant, category, date) — see the vector search guide.
EVALUATE retrieval with real queries and labeled relevance (Recall@k), not by eyeballing a few results.
LIMITS: embeddings capture overall semantic/visual similarity, not fine detail. They can miss small text in an image, exact counts, or subtle defects —
  pair them with OCR or detection when those matter.
PRIVACY: embeddings derived from sensitive images (faces, medical, documents) are sensitive data.

7. Azure AI Vision

Microsoft's managed vision services

AZURE AI VISION — IMAGE ANALYSIS 4.0   General image understanding: CAPTIONS (a sentence describing the image), DENSE CAPTIONS (regions),
                                       TAGS, OBJECT DETECTION, PEOPLE DETECTION, SMART CROPS, and READ (OCR). One API, choose features per call.

AZURE AI VISION — MULTIMODAL           Image and text embeddings for vector search (Section 6).
EMBEDDINGS

AZURE AI DOCUMENT INTELLIGENCE         Document OCR, layout, prebuilt extractors (invoice, receipt, ID, ...) and custom models (Section 4).

AZURE AI FACE                          Face detection and, for approved use cases, verification/identification (Section 5; Limited Access applies).

AZURE OPENAI (VISION-CAPABLE MODELS)   Multimodal chat models that understand images (Section 10).

AZURE AI CONTENT SAFETY                Detects harmful content in images and text — useful for moderating user uploads.

CONTAINERS                             Some Vision and Document Intelligence capabilities are available as containers you can run in your own
                                       environment for data-residency or disconnected scenarios (check which features and licensing apply).

Availability, regions, and feature maturity vary, and Microsoft has retired or announced retirement of older offerings (earlier Computer Vision API versions, Custom Vision, and some restricted Face features among them). Check the current service documentation and retirement notices before depending on any feature.

Image Analysis 4.0 in C

// dotnet add package Azure.AI.Vision.ImageAnalysis
using Azure;
using Azure.AI.Vision.ImageAnalysis;

var client = new ImageAnalysisClient(new Uri(endpoint), new AzureKeyCredential(key));

ImageAnalysisResult result = await client.AnalyzeAsync(
    BinaryData.FromBytes(imageBytes),
    VisualFeatures.Caption | VisualFeatures.Tags | VisualFeatures.Objects | VisualFeatures.People | VisualFeatures.Read,
    new ImageAnalysisOptions { GenderNeutralCaption = true, Language = "en" });

Console.WriteLine($"Caption: {result.Caption.Text} ({result.Caption.Confidence:P0})");

foreach (DetectedTag tag in result.Tags.Values)
    Console.WriteLine($"Tag: {tag.Name} ({tag.Confidence:P0})");

foreach (DetectedObject obj in result.Objects.Values)
    Console.WriteLine($"Object: {obj.Tags[0].Name} ({obj.Tags[0].Confidence:P0}) at X={obj.BoundingBox.X}, Y={obj.BoundingBox.Y}, W={obj.BoundingBox.Width}, H={obj.BoundingBox.Height}");

foreach (DetectedPerson person in result.People.Values.Where(p => p.Confidence > 0.6))
    Console.WriteLine($"Person at {person.BoundingBox.X},{person.BoundingBox.Y} ({person.Confidence:P0})");

You can pass a Uri instead of bytes when the service can fetch the image itself. Request only the visual features you need — each feature adds to the response and the bill, and some (such as captions) are available only in certain regions.

Authentication, networking, and operations

AUTH            Prefer Microsoft Entra ID (DefaultAzureCredential, managed identity) over API keys; if you use keys, keep them in Key Vault.
                (The SDK clients accept either.)
NETWORKING      Private endpoints and virtual-network restrictions are available for enterprise deployments.
LIMITS          Image size and dimension limits, supported formats, and requests-per-second quotas apply — resize before upload, and handle
                HTTP 429 with backoff (see AI APIs with .NET, Sections 10-11). The SDK clients include retry policies for transient errors.
COST            Billed per transaction (per feature, per image). Cache results by image hash, don't re-analyze unchanged images, and request only
                needed features.
DATA            Understand what the service retains and where it processes data; images can be sensitive. Use containers or private networking if policy requires.

Wire it through your own abstraction

public interface IImageAnalyzer
{
    Task<ImageSummary> AnalyzeAsync(byte[] image, CancellationToken ct);
}

public record ImageSummary(string Caption, IReadOnlyList<string> Tags, IReadOnlyList<(string Label, double Confidence, int X, int Y, int W, int H)> Objects, string Text);

Hide the Azure types behind your own interface so you can swap in a self-hosted model or a different provider, add caching and resilience in one place, and unit test with a fake — the same decoupling argument made for AI services in Microsoft.Extensions.AI.

8. OpenCV Integration

OpenCV: the classic computer vision toolkit

OpenCV is a mature, high-performance library for image and video processing and "classical" computer vision: filtering, geometric transforms, edge and contour detection, feature matching, camera calibration, video I/O, background subtraction, tracking, and a DNN module that can run neural networks. In .NET you use it through OpenCvSharp (the most popular open-source wrapper) or Emgu CV (another wrapper with its own licensing terms to review).

Setup

dotnet add package OpenCvSharp4
dotnet add package OpenCvSharp4.runtime.win          # native OpenCV binaries — choose the runtime package for YOUR platform:
                                                     #   .runtime.win, .runtime.ubuntu.20.04-x64, .runtime.linux-arm, .runtime.osx ...
dotnet add package OpenCvSharp4.Extensions            # conversions to/from System.Drawing bitmaps (Windows-oriented)

The managed package is a wrapper; native binaries must be present at runtime, which is the main deployment gotcha (especially in Docker/Linux — install the matching runtime package and any required system libraries, and test in the actual container image).

The Mat and the dispose rule

using OpenCvSharp;

// Mat is OpenCV's image/matrix type. It owns NATIVE memory that the .NET garbage collector cannot see —
// ALWAYS dispose it (using), especially in loops and servers, or you will leak memory.
using Mat image = Cv2.ImRead("photo.jpg", ImreadModes.Color);       // loads as BGR (not RGB!)
using Mat gray  = new();
Cv2.CvtColor(image, gray, ColorConversionCodes.BGR2GRAY);

using Mat blurred = new();
Cv2.GaussianBlur(gray, blurred, new Size(5, 5), 0);

using Mat edges = new();
Cv2.Canny(blurred, edges, threshold1: 50, threshold2: 150);

Cv2.ImWrite("edges.png", edges);

Colors are BGR — the classic bug

OpenCV stores color images as BLUE-GREEN-RED, while most deep-learning models (and most other imaging libraries, including ImageSharp and
SkiaSharp) use RGB. Feed an unconverted BGR image to an RGB-trained model and accuracy silently drops. Convert with
Cv2.CvtColor(src, dst, ColorConversionCodes.BGR2RGB) at the boundary.

Common operations

Cv2.Resize(image, resized, new Size(640, 480));                                   // resize
Cv2.Rotate(image, rotated, RotateFlags.Rotate90Clockwise);                         // rotate
using var roi = new Mat(image, new Rect(100, 50, 200, 200));                       // crop (a view into the same data)

// Find contours (shapes) in a binary image
Cv2.FindContours(binary, out Point[][] contours, out HierarchyIndex[] hierarchy, RetrievalModes.External, ContourApproximationModes.ApproxSimple);
foreach (var contour in contours.Where(c => Cv2.ContourArea(c) > 500))
    Cv2.Rectangle(image, Cv2.BoundingRect(contour), Scalar.Lime, 2);

// Draw and label detections from Section 3
foreach (var d in detections)
{
    var rect = new Rect((int)d.Box.X, (int)d.Box.Y, (int)d.Box.Width, (int)d.Box.Height);
    Cv2.Rectangle(image, rect, Scalar.Red, 2);
    Cv2.PutText(image, $"{d.Label} {d.Confidence:P0}", new Point(rect.X, Math.Max(15, rect.Y - 5)),
                HersheyFonts.HersheySimplex, 0.6, Scalar.Red, 2);
}

Moving between OpenCV, bytes, and other libraries

// Encode/decode in memory (no temp files) — the way to move images over HTTP or through queues
Cv2.ImEncode(".jpg", image, out byte[] jpegBytes);
using Mat decoded = Cv2.ImDecode(jpegBytes, ImreadModes.Color);

// To feed a neural-network input tensor from a Mat, copy to a managed array and convert BGR -> RGB deliberately (not by accident)

Video: frame loops

using var capture = new VideoCapture("traffic.mp4");          // a file path, an RTSP/HTTP URL, or a camera index (0)
using var frame = new Mat();

int frameIndex = 0;
while (capture.Read(frame) && !frame.Empty())
{
    // Detection is expensive: run it every Nth frame (and track in between) instead of every frame
    if (frameIndex++ % 5 == 0)
    {
        // convert and run a detector (Section 3) ...
    }
}

VIDEO PRACTICALITIES
  - Decode on a worker thread; keep a small bounded queue between capture, inference, and output so a slow model drops frames instead of
    accumulating latency and memory.
  - Process every Nth frame or at reduced resolution for real-time budgets.
  - Use hardware decoding and GPU inference when throughput matters.
  - Reconnect automatically for network streams; they drop.
  - Watch resource use: each frame is a large native allocation.

OpenCV's DNN module

using OpenCvSharp.Dnn;

using Net net = CvDnn.ReadNetFromOnnx("model.onnx");
using Mat blob = CvDnn.BlobFromImage(image, scaleFactor: 1 / 255.0, size: new Size(640, 640), mean: new Scalar(), swapRB: true, crop: false);
net.SetInput(blob);
using Mat output = net.Forward();

OpenCV can run many ONNX models directly (convenient for simple deployments, and for models such as YuNet face detection). For broader operator support, GPU execution providers, and performance tuning, ONNX Runtime is usually the stronger inference engine; many teams use OpenCV for image I/O and preprocessing, ONNX Runtime for inference.

When to reach for OpenCV

VIDEO AND CAMERAS            capture, decode, encode, streams
GEOMETRIC / CLASSICAL CV     perspective correction, deskewing, contour and shape analysis, feature matching, calibration, background subtraction
PREPROCESSING FOR OCR        grayscale, thresholding, denoising, deskew (Section 4)
FAST NATIVE IMAGE OPS        heavy per-pixel work where native performance matters
Prefer a managed library (Section 9) for simple resizing, format conversion, and drawing in web back ends — fewer native dependencies to deploy.

9. Image Processing APIs

Choosing a library for decoding, transforming, drawing, and encoding

Before and after any model, you decode images, orient and resize them, convert formats, and sometimes draw annotations. .NET offers several libraries, with important differences in platform support and licensing.

SixLabors.ImageSharp      Fully managed, cross-platform, modern API, no native dependencies, good for web servers and containers.
                          Drawing lives in the separate ImageSharp.Drawing package. NOTE: it uses a "split license" — free for open-source and
                          qualifying small organizations, but commercial use may require a paid license; check the current terms.

SkiaSharp                 Bindings to Google's Skia graphics engine; fast, widely used (MAUI, Avalonia), strong 2D drawing and text rendering. Native
                          binaries per platform; permissive MIT license.

System.Drawing.Common     GDI+ based. On .NET 6+ it is supported on WINDOWS ONLY. Don't use it for cross-platform or Linux-container services.

Magick.NET                Wrapper over ImageMagick: enormous format support and many operations; native dependencies; Apache-style license.

NetVips                   Wrapper over libvips: very fast and memory-efficient for large images and pipelines; native dependency.

OpenCvSharp               Computer vision and video, with image I/O and drawing (Section 8). BGR channel order.

Windows imaging APIs      WIC/Windows.Graphics.Imaging for Windows-only apps.

ImageSharp: the everyday operations

using SixLabors.ImageSharp;
using SixLabors.ImageSharp.Formats.Jpeg;
using SixLabors.ImageSharp.Processing;

// Load -> fix orientation -> resize (never enlarge) -> strip metadata -> save
using Image image = await Image.LoadAsync(inputStream);

image.Mutate(x => x
    .AutoOrient()                                                                  // apply the EXIF rotation phones store instead of rotating pixels
    .Resize(new ResizeOptions { Size = new Size(1600, 1600), Mode = ResizeMode.Max }));   // fit within 1600x1600, keep aspect ratio

image.Metadata.ExifProfile = null;                                                 // remove EXIF (GPS location, device info) before storing or sharing

await image.SaveAsJpegAsync(outputStream, new JpegEncoder { Quality = 85 });

Cross-cutting image gotchas

EXIF ORIENTATION   Phone photos are often stored sideways with an orientation flag. If you ignore it, the model sees a rotated image (and OCR fails).
                   Apply orientation (AutoOrient) BEFORE inference.

EXIF PRIVACY       Photos carry GPS coordinates, timestamps, and device details. Strip metadata before storing or publishing images you didn't create.

COLOR ORDER        RGB (ImageSharp, SkiaSharp, most models) vs BGR (OpenCV). Convert deliberately.

COLOR SPACES       Some images are CMYK, grayscale, 16-bit, or have alpha. Convert to a known format (e.g. Rgb24) before building tensors.

RESIZE QUALITY     Choose a good resampler (bicubic/Lanczos) for downscaling; the choice can measurably affect model accuracy, and it should match
                   training if the model is sensitive.

MEMORY             A 50-megapixel image decodes to ~150 MB of pixels. Cap dimensions BEFORE full decode (read just the header with Image.Identify) and
                   resize early.

THREAD SAFETY      Image/Mat objects are not thread-safe; don't share one across concurrent requests.

FORMAT SUPPORT     JPEG, PNG, WebP, GIF, BMP, TIFF are common; HEIC (iPhone) and RAW formats often need extra libraries — plan for them if users upload from phones.

Draw on images: ImageSharp.Drawing and SkiaSharp

// SkiaSharp: draw labeled boxes on a decoded bitmap
using SkiaSharp;

using SKBitmap bitmap = SKBitmap.Decode(imageBytes);
using var canvas = new SKCanvas(bitmap);
using var boxPaint = new SKPaint { Color = SKColors.Red, Style = SKPaintStyle.Stroke, StrokeWidth = 3 };

foreach (var d in detections)
    canvas.DrawRect(d.Box.X, d.Box.Y, d.Box.Width, d.Box.Height, boxPaint);

using var image = SKImage.FromBitmap(bitmap);
using var data = image.Encode(SKEncodedImageFormat.Jpeg, 85);
await File.WriteAllBytesAsync("annotated.jpg", data.ToArray());

Validate uploaded images — they are untrusted input

// Read dimensions from the header WITHOUT decoding the whole image, and reject abusive inputs early
ImageInfo info = await Image.IdentifyAsync(stream);                  // throws on unrecognized formats
if (info.Width > 10_000 || info.Height > 10_000 || (long)info.Width * info.Height > 50_000_000)
    return Results.BadRequest("Image dimensions are too large.");
stream.Position = 0;

Treat image uploads like any untrusted file: enforce a maximum byte size (and request-body limits); verify the actual format from content, not the file extension
or Content-Type header; limit decoded dimensions (decompression bombs — a tiny file that expands to gigabytes of pixels); process in a resource-limited
context; re-encode to a safe format rather than storing originals blindly; and scan for malware where policy requires. Keep image-decoding libraries updated —
image codecs are a historically common source of vulnerabilities.

10. Multimodal LLMs vs. Classical Computer Vision

Vision-capable chat models are now a mainstream option

Modern chat models accept images alongside text. Through Microsoft.Extensions.AI (see this series' Microsoft.Extensions.AI guide), sending an image is just another content item in a message:

using Microsoft.Extensions.AI;

var message = new ChatMessage(ChatRole.User,
[
    new TextContent("Extract the vendor name, invoice number, date, and total from this invoice. Respond as JSON."),
    new DataContent(imageBytes, "image/jpeg"),
]);

ChatResponse response = await chatClient.GetResponseAsync([message]);

// Or get a typed result directly
public record InvoiceFields(string Vendor, string InvoiceNumber, string Date, decimal Total);
ChatResponse<InvoiceFields> typed = await chatClient.GetResponseAsync<InvoiceFields>([message]);
InvoiceFields invoice = typed.Result;                          // validate it — shape is guaranteed, truth is not

What multimodal LLMs are good at

OPEN-ENDED QUESTIONS     "Is anything unsafe in this photo?" "Describe what happened in this screenshot." — no model training or fixed label set.
ZERO-SHOT FLEXIBILITY    New categories or extraction schemas by changing the prompt, not retraining.
MESSY DOCUMENTS          Irregular layouts, handwriting, receipts photographed at angles, charts and diagrams.
REASONING OVER IMAGES    Combining visual content with text instructions and context.
RAPID PROTOTYPING        Proof of concept in an afternoon.

Where they fall short

HALLUCINATION         They can confidently invent text, numbers, or objects that aren't in the image. For OCR-like extraction this is dangerous: a plausible but
                      wrong total looks exactly like a right one.
PRECISE LOCALIZATION  Bounding boxes, pixel-accurate positions, and exact counting of many objects are unreliable compared with dedicated detectors.
CONSISTENCY           Non-deterministic; the same image can yield different outputs.
COST AND LATENCY      Each image consumes many tokens and takes seconds — painful at video frame rates or millions of images.
THROUGHPUT            Rate limits and per-call pricing don't suit high-volume, real-time pipelines.
PRIVACY               Images are sent to a hosted model; sensitive content (faces, IDs, medical images) raises data-protection questions.
LIMITS                Image size and count per request, and resolution that may downscale small text.

Choosing

Hybrid pipelines are often best

Classical + LLM, each doing what it's best at:
  - OCR / Document Intelligence extracts the text and fields WITH confidence; an LLM interprets or normalizes them — or only handles the low-confidence cases.
  - A detector finds and crops regions; an LLM describes or reads each crop.
  - A cheap classifier screens images; only the uncertain ones go to the expensive multimodal model.
  - An LLM proposes; deterministic code VALIDATES (totals add up, dates parse, values within ranges).

All the cautions from AI Agents and LLM Application Development apply: treat text inside images as untrusted (images can contain visual prompt injection — instructions written into a picture), validate outputs, and keep a human in the loop for consequential decisions.

11. Building a Vision Service in ASP.NET Core

A detection endpoint, with the production concerns included

// Program.cs
builder.Services.AddSingleton(sp => new ObjectDetector("models/yolov8n.onnx", CocoLabels.All));   // load the model ONCE

builder.Services.AddRateLimiter(o => o.AddPolicy("vision", ctx => RateLimitPartition.GetFixedWindowLimiter(
    ctx.User.Identity?.Name ?? "anonymous", _ => new FixedWindowRateLimiterOptions { PermitLimit = 30, Window = TimeSpan.FromMinutes(1) })));

builder.WebHost.ConfigureKestrel(k => k.Limits.MaxRequestBodySize = 10_000_000);                   // cap upload size

var inference = new SemaphoreSlim(Environment.ProcessorCount);                                     // bound concurrent CPU-heavy inference

app.MapPost("/detect", async (IFormFile file, ObjectDetector detector, CancellationToken ct) =>
{
    if (file.Length is 0 or > 10_000_000) return Results.BadRequest("Image must be between 1 byte and 10 MB.");

    await using Stream stream = file.OpenReadStream();

    ImageInfo info;
    try { info = await Image.IdentifyAsync(stream, ct); }                                          // verify it's a real image; read dimensions cheaply
    catch (Exception) { return Results.BadRequest("Unsupported or corrupt image."); }

    if ((long)info.Width * info.Height > 50_000_000) return Results.BadRequest("Image dimensions are too large.");
    stream.Position = 0;

    using Image<Rgb24> image = await Image.LoadAsync<Rgb24>(stream, ct);
    image.Mutate(x => x.AutoOrient());                                                             // honor EXIF rotation BEFORE inference

    await inference.WaitAsync(ct);
    try
    {
        var detections = await Task.Run(() => detector.Detect(image), ct);                         // keep the request thread free; bound the work
        return Results.Ok(detections.Select(d => new { d.Label, d.Confidence, d.Box.X, d.Box.Y, d.Box.Width, d.Box.Height }));
    }
    finally { inference.Release(); }
})
.RequireAuthorization()
.RequireRateLimiting("vision")
.DisableAntiforgery();                                                                             // an API endpoint, not a browser form post

Architecture and performance considerations

LOAD MODELS ONCE            Register the inference session as a singleton; loading a model per request is slow and memory-hungry. InferenceSession.Run
                            is safe to call concurrently; image objects are not shared.

WARM UP                     The first inference is slower (graph optimization, allocations). Run a dummy inference at startup so users don't pay for it.

BOUND CONCURRENCY           CPU/GPU inference is resource-intensive; limit parallel work (SemaphoreSlim) and queue the rest rather than thrashing.

ASYNC AND BACKGROUND        For slow or bulk jobs (video, large batches, heavy OCR), accept the upload, enqueue a job, return an ID, and let a background worker process it
                            (BackgroundService / queue / Azure Functions), with status polling or a webhook/SignalR notification.

GPU                         Microsoft.ML.OnnxRuntime.Gpu (CUDA) or .DirectML (Windows) can accelerate inference substantially; build and test GPU container images
                            deliberately (drivers, CUDA/cuDNN versions).

MODEL SIZE VS. SPEED        Smaller or quantized models (INT8) and lower input resolutions cut latency and memory at some accuracy cost; measure the trade-off.

BATCHING                    Batching several images into one inference call raises GPU throughput at the cost of per-request latency.

CACHE                       Key results by a hash of the image bytes (plus model version) — repeated images are common (duplicates, retries, shared files).

MODEL MANAGEMENT            Version models like code; store them in blob storage or the image, record the model version with each result, and use a rolling
                            rollout so you can compare and roll back.

OBSERVABILITY               Log latency, input sizes, confidence distributions, and error rates; export traces/metrics (OpenTelemetry). Drift in confidence
                            distributions is an early warning that production images have changed.

DOCKER                      Include native dependencies (OpenCV runtime packages, ONNX Runtime native libs) and test the actual image; keep images small.

Security and privacy for image services

UNTRUSTED UPLOADS       Size/dimension limits, format verification, resource-limited processing, updated codecs (Section 9).
AUTHORIZATION           Users may only analyze and retrieve THEIR images/results; scope storage and queries per user/tenant.
SENSITIVE CONTENT       Images may contain faces, IDs, medical or financial documents. Minimize retention, encrypt at rest and in transit, restrict who and what
                        can see them, and be explicit about when images leave your environment (cloud APIs, LLMs).
EXIF AND METADATA       Strip location and device data before storing or sharing.
MODERATION              If users upload content others will see, screen it (for example with Azure AI Content Safety).
ADVERSARIAL INPUTS      Models can be fooled by crafted images (adversarial examples); don't rely on vision output alone for security-critical decisions. Images can
                        also carry hidden instructions aimed at multimodal LLMs.
COSTS AND ABUSE         Rate limit, cap sizes, and authenticate — an open image-analysis endpoint (especially one calling a paid API) is a cost-abuse target.

12. Evaluating Vision Systems

Pick metrics that match the task — then test on realistic data

CLASSIFICATION    Accuracy (careful with imbalance), precision, recall, F1, per-class results, confusion matrix, calibration of confidence scores.
                  (See AI/ML Fundamentals, Section 9.)

OBJECT DETECTION  IoU between predicted and true boxes; a detection is correct if the label matches and IoU exceeds a threshold (commonly 0.5).
                    PRECISION = correct detections / all detections;   RECALL = correct detections / all true objects.
                    AP (average precision) per class summarizes the precision-recall curve; mAP averages across classes (and sometimes IoU thresholds).
                  Also check counting error and per-size performance (small objects are hardest).

OCR               Character error rate (CER) and word error rate (WER) for text; FIELD-LEVEL accuracy and per-field confidence calibration for extraction;
                  end-to-end "is the final record correct?" rate.

FACE VERIFICATION FAR and FRR at the chosen threshold, ROC curves, and results BROKEN DOWN BY DEMOGRAPHIC GROUP and imaging condition.

EMBEDDINGS /      Recall@k and MRR on labeled query-to-relevant-image pairs (see Embeddings & Vector Search and RAG, Section 12).
VISUAL SEARCH

LLM VISION        Task-specific accuracy against ground truth, plus hallucination checks (does the extracted value actually appear in the image?),
                  consistency across repeated runs, and cost/latency.

Data practices that make evaluation honest

REPRESENTATIVE TEST SET    Collect images from the REAL production environment: your cameras, lighting, devices, angles, resolutions, and edge cases (blur, glare,
                           occlusion, odd orientations). Models validated on clean benchmark photos routinely disappoint in the field.
SPLIT BY SOURCE            Keep near-duplicates (frames from one video, shots of the same object, pages of one document) on one side of the split.
CONSISTENT LABELS          Ambiguous or noisy labels cap achievable accuracy. Write labeling guidelines, measure inter-annotator agreement, and review disputed cases.
STRATIFY AND SLICE         Report results by condition (day/night, indoor/outdoor, device type), by class, and by object size — averages hide failures.
FAIRNESS                   Where people appear, evaluate across demographic groups; large gaps are a defect even when the average looks good.
NEGATIVE EXAMPLES          Include images with NONE of the target objects, and out-of-distribution images, to measure false positives and unknown handling.
FROZEN TEST SET            Don't tune on it. Keep a held-out set untouched until the end, and refresh it as the world changes.

In production

CONFIDENCE DISTRIBUTIONS   Track them over time; shifts signal that production images no longer match what the model was validated on.
HUMAN REVIEW QUEUE         Route low-confidence or high-stakes results to people; use their corrections as new labeled data.
SAMPLED AUDIT              Regularly review random samples of automated decisions, not just the flagged ones.
FEEDBACK LOOP              Capture errors and overrides, label them, and fold them into the next training and evaluation set.
VERSIONED REGRESSION SUITE Re-run your evaluation set whenever you change models, thresholds, preprocessing, or a provider's model version.
COST AND LATENCY BUDGETS   Track p50/p95 latency and per-image cost alongside accuracy.

13. Choosing the Right Approach

START WITH THE REQUIREMENTS, NOT THE MODEL
  What decision does this feed? What does a mistake cost? Is it real-time? How many images per day? Do images contain sensitive data? Must it run offline or
  inside a boundary? Are the categories common or specific to you?

THEN WALK THE LADDER, STOPPING AS EARLY AS POSSIBLE
  1. A managed API handles it (Azure AI Vision, Document Intelligence)        -> build in days; pay per call
  2. A multimodal LLM handles it well enough                                  -> flexible and fast to build; check cost, latency, and hallucination risk
  3. A pre-trained ONNX model handles it                                      -> run locally; no per-call fee; you own pre/postprocessing
  4. You need custom accuracy on your own data                                -> labeled dataset, transfer learning or fine-tuning, evaluation, MLOps

THEN HARDEN
  Evaluate on realistic images, set thresholds and a human-review path, add validation, secure the pipeline, monitor drift, and plan for model and service lifecycle changes.

14. Common Pitfalls

Quick Reference Table

Conclusion

Computer vision in .NET is no longer a research detour: a managed API can read an invoice or caption a photo in one call, an ONNX model can detect objects at video rates inside your own process, ML.NET can fine-tune a classifier on a few hundred of your images, embeddings turn "find me something that looks like this" into a vector search, and a multimodal LLM can answer open-ended questions about almost anything you show it. The real skill is choosing among those options deliberately — by accuracy needs, volume, latency, cost, data sensitivity, and the cost of a mistake — and then engineering the unglamorous parts correctly.

Those parts are where most projects succeed or fail. Preprocessing must match what the model was trained on — size, channel order, scaling, orientation — or accuracy collapses without any error message. Postprocessing must decode the model's actual output format, apply thresholds and non-maximum suppression, and map coordinates back to the original image. Native resources must be disposed, models loaded once, concurrency bounded, uploads treated as untrusted input, and metadata stripped. Outputs from OCR and multimodal models must be validated against business rules and routed to humans when confidence is low, because plausible-looking wrong answers are the dangerous kind. Face recognition deserves a legal and ethical decision before a technical one; detection-only or carefully consented, locally matched designs are usually the safer path. And above all, evaluate on images that look like production — sliced by condition, class, and (where people appear) demographic group — because a benchmark score on clean photos tells you very little about how the system will behave in the field. Build with those habits and computer vision becomes a reliable component of your .NET applications rather than a demo that stops working the moment real images arrive.

Found this useful? Feel free to star the repo, open an issue with corrections, or share the "the model worked perfectly until we noticed phone photos were all sideways" story that made the case for AutoOrient and representative test data click better than any benchmark.