如何用 Go 擷取 Mastodon 資料：跨伺服器搜尋個人檔案、貼文、標籤與趨勢（2026）

Scraping Mastodon is unusual in a good way: you don't need to parse HTML at all. Every Mastodon server exposes the same public REST API, and most read endpoints answer without a token. The hard part isn't getting data. It's that there is no single "Mastodon". There are thousands of servers, each with its own partial view of the network, its own rate limits, and its own settings.

In this post I'll build a small Mastodon scraper in Go, standard library only, that:

1. Looks up a profile on the account's home server
2. Pages through a user's original posts for the last N days and computes engagement
3. Collects a hashtag from several servers and de-duplicates the results
4. Reads trending hashtags per server
5. Writes everything to CSV

Everything below was run on October 26, 2026, and the output is real.

Before you start: read the server's rules and terms, keep request rates low, and respect people's choices. Mastodon accounts can set noindex (opt out of search engines) and indexable flags; the code below skips noindex accounts. Don't build profiles of individual people, and don't republish their posts out of context.

How the Mastodon API looks from a scraper's point of view

All the endpoints in this post are plain GET requests that return JSON:

The last two rows are the first surprise. On mastodon.social, the public timeline answers HTTP 422 with {"error":"This method requires an authenticated user"}, and status search returns an empty statuses array when you're logged out. Server admins can turn these on or off, so check them on each server you target.

Pagination isn't in the JSON body. It's in the Link response header, like GitHub's API:

Link: <https://mastodon.social/api/v1/accounts/1/statuses?max_id=115...>; rel="next", <...>; rel="prev"

Rate limits are in headers too. Unauthenticated requests on mastodon.social got x-ratelimit-limit: 300 per five-minute window, with x-ratelimit-remaining and an x-ratelimit-reset timestamp.

Step 1: A polite HTTP helper

mkdir mastoscraper && cd mastoscraper
go mod init mastoscraper

The helper sends a real User-Agent (put your own contact URL in it), retries on network errors and 5xx, sleeps until the reset time on a 429, and slows down on its own when fewer than 20 requests are left in the window. It returns the response headers, because we need Link for pagination.

package main

import (
    "encoding/csv"
    "encoding/json"
    "errors"
    "flag"
    "fmt"
    "html"
    "net/http"
    "net/url"
    "os"
    "regexp"
    "sort"
    "strconv"
    "strings"
    "time"
)

const userAgent = "mastoscraper/0.1 (+https://example.com/contact)"

var client = &http.Client{Timeout: 20 * time.Second}

// getJSON fetches a URL, decodes JSON into out, and returns the response
// headers so callers can read Link (pagination) and X-RateLimit-*.
func getJSON(u string, out any) (http.Header, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, _ := http.NewRequest("GET", u, nil)
        req.Header.Set("User-Agent", userAgent)
        req.Header.Set("Accept", "application/json")
        res, err := client.Do(req)
        if err != nil {
            time.Sleep(time.Duration(attempt+1) * 2 * time.Second)
            continue
        }
        if res.StatusCode == 429 {
            wait := untilReset(res.Header)
            fmt.Fprintf(os.Stderr, "429 from %s, sleeping %s\n", res.Request.URL.Host, wait)
            res.Body.Close()
            time.Sleep(wait)
            continue
        }
        if res.StatusCode >= 500 {
            res.Body.Close()
            time.Sleep(time.Duration(attempt+1) * 3 * time.Second)
            continue
        }
        if res.StatusCode != 200 {
            res.Body.Close()
            return res.Header, fmt.Errorf("GET %s: HTTP %d", u, res.StatusCode)
        }
        err = json.NewDecoder(res.Body).Decode(out)
        res.Body.Close()
        // Be polite even when we have budget left.
        if rem, _ := strconv.Atoi(res.Header.Get("X-RateLimit-Remaining")); rem > 0 && rem < 20 {
            time.Sleep(untilReset(res.Header))
        }
        return res.Header, err
    }
    return nil, errors.New("gave up after retries: " + u)
}

func untilReset(h http.Header) time.Duration {
    if t, err := time.Parse(time.RFC3339Nano, h.Get("X-RateLimit-Reset")); err == nil {
        if d := time.Until(t); d > 0 {
            return d + time.Second
        }
    }
    return 30 * time.Second
}

// nextURL pulls the rel="next" URL out of a Link header like
// <https://host/api/v1/...?max_id=123>; rel="next", <...>; rel="prev"
var linkNext = regexp.MustCompile(`<([^>]+)>;\s*rel="next"`)

func nextURL(h http.Header) string {
    m := linkNext.FindStringSubmatch(h.Get("Link"))
    if m == nil {
        return ""
    }
    return m[1]
}

Step 2: Types and small helpers

Mastodon returns post bodies as HTML (<p>, links, mention spans), so plainText strips tags and unescapes entities. splitHandle accepts @user@server, user@server, or a profile URL.

type Account struct {
    ID             string `json:"id"`
    Acct           string `json:"acct"`
    DisplayName    string `json:"display_name"`
    URL            string `json:"url"`
    FollowersCount int    `json:"followers_count"`
    FollowingCount int    `json:"following_count"`
    StatusesCount  int    `json:"statuses_count"`
    LastStatusAt   string `json:"last_status_at"`
    Bot            bool   `json:"bot"`
    Indexable      *bool  `json:"indexable"`
    Noindex        *bool  `json:"noindex"`
}

type Status struct {
    ID              string    `json:"id"`
    URI             string    `json:"uri"`
    URL             string    `json:"url"`
    CreatedAt       time.Time `json:"created_at"`
    Content         string    `json:"content"`
    Language        string    `json:"language"`
    RepliesCount    int       `json:"replies_count"`
    ReblogsCount    int       `json:"reblogs_count"`
    FavouritesCount int       `json:"favourites_count"`
    Account         Account   `json:"account"`
    Reblog          *Status   `json:"reblog"`
    Tags            []struct {
        Name string `json:"name"`
    } `json:"tags"`
}

var tagRe = regexp.MustCompile(`<[^>]+>`)

func plainText(h string) string {
    h = strings.ReplaceAll(h, "</p><p>", "\n\n")
    h = strings.ReplaceAll(h, "<br />", "\n")
    return strings.TrimSpace(html.UnescapeString(tagRe.ReplaceAllString(h, "")))
}

// splitHandle turns "@user@server", "user@server" or a profile URL into (user, server).
func splitHandle(handle string) (string, string, error) {
    if strings.HasPrefix(handle, "https://") {
        u, err := url.Parse(handle)
        if err != nil {
            return "", "", err
        }
        return strings.TrimPrefix(strings.Trim(u.Path, "/"), "@"), u.Host, nil
    }
    parts := strings.Split(strings.TrimPrefix(handle, "@"), "@")
    if len(parts) != 2 {
        return "", "", fmt.Errorf("want user@server, got %q", handle)
    }
    return parts[0], parts[1], nil
}

Step 3: Profiles and posts, from the home server

Any server can look up any account it knows about. mastodon.social will happily answer lookup?acct=golang@hachyderm.io. But that answer is mastodon.social's cached copy, and it's only as fresh as the last time mastodon.social refreshed it. In my checks today the numbers matched, but nothing guarantees that, and copies of posts on other servers definitely lag behind on favourites and boosts. So the code asks the account's own server.

exclude_replies=true&exclude_reblogs=true keeps only original posts, limit=40 is the maximum page size, and the loop stops when posts get older than the cutoff. Posts come back newest first, so it can stop early instead of paging through years of history.

// lookup asks the account's HOME server, not mastodon.social, so the counts are authoritative.
func lookup(handle string) (Account, string, error) {
    user, server, err := splitHandle(handle)
    if err != nil {
        return Account{}, "", err
    }
    var a Account
    _, err = getJSON(fmt.Sprintf("https://%s/api/v1/accounts/lookup?acct=%s", server, url.QueryEscape(user)), &a)
    return a, server, err
}

func userPosts(handle string, maxPages int, since time.Time) ([]Status, error) {
    a, server, err := lookup(handle)
    if err != nil {
        return nil, err
    }
    if a.Noindex != nil && *a.Noindex {
        return nil, fmt.Errorf("%s has opted out of search indexing (noindex); skipping", handle)
    }
    next := fmt.Sprintf("https://%s/api/v1/accounts/%s/statuses?limit=40&exclude_replies=true&exclude_reblogs=true", server, a.ID)
    var all []Status
    for page := 0; next != "" && page < maxPages; page++ {
        var batch []Status
        h, err := getJSON(next, &batch)
        if err != nil {
            return all, err
        }
        stop := false
        for _, s := range batch {
            if s.CreatedAt.Before(since) {
                stop = true
                break
            }
            all = append(all, s)
        }
        if stop || len(batch) == 0 {
            break
        }
        next = nextURL(h)
    }
    return all, nil
}

$ ./mastoscraper profile @Gargron@mastodon.social
Eugen Rochko (Gargron) on mastodon.social
  followers 383376 · following 764 · posts 82664 · last post 2026-10-26 · bot=false

$ ./mastoscraper posts -days 30 -pages 3 @Gargron@mastodon.social
34 original posts in the last 30 days
avg favourites 184.7 · avg boosts 61.3
  1012 ★ 451 ⟳  Heh. I misremembered the date--thought it was today, but loo…
   961 ★ 450 ⟳  "AI" is a failed technology, in the sense that it's neither …
   468 ★  95 ⟳  #SilentSunday

Note that last_status_at is a date, not a timestamp. Mastodon rounds it on purpose.

Step 4: One hashtag, several servers, no duplicates

This is where federation matters. A hashtag timeline on a server only contains posts that server has seen: posts by its own users, plus posts from accounts someone on that server follows. #golang on mastodon.social, fosstodon.org, and hachyderm.io are three different, overlapping lists.

So the code reads the tag from each server and de-duplicates by uri. The uri is the post's canonical ActivityPub id, and it's the same on every server. The id field is not: each server assigns its own local id to its copy of a post. When the same post shows up more than once, it keeps the copy with the highest counts, since remote copies are the ones that lag.

// tagAcrossServers reads the same hashtag from several servers and de-duplicates by URI,
// because each server only sees the posts that federated to it.
func tagAcrossServers(tag string, servers []string, maxPages int) (map[string]Status, map[string]int) {
    seen := map[string]Status{}
    perServer := map[string]int{}
    for _, srv := range servers {
        next := fmt.Sprintf("https://%s/api/v1/timelines/tag/%s?limit=40", srv, url.PathEscape(tag))
        for page := 0; next != "" && page < maxPages; page++ {
            var batch []Status
            h, err := getJSON(next, &batch)
            if err != nil {
                fmt.Fprintf(os.Stderr, "%s: %v\n", srv, err)
                break
            }
            for _, s := range batch {
                perServer[srv]++
                prev, ok := seen[s.URI]
                // Keep the copy with the highest counts; remote copies lag behind.
                if !ok || s.FavouritesCount+s.ReblogsCount > prev.FavouritesCount+prev.ReblogsCount {
                    seen[s.URI] = s
                }
            }
            if len(batch) == 0 {
                break
            }
            next = nextURL(h)
        }
    }
    return seen, perServer
}

The acct field has its own trap. It's relative to the server you asked: an account local to hachyderm.io shows up as plain golang when you ask hachyderm.io, and as golang@hachyderm.io everywhere else. If you mix servers, normalize it:

// fullAcct makes "user" (local to whichever server we asked) into "user@server".
func fullAcct(a Account) string {
    if strings.Contains(a.Acct, "@") {
        return a.Acct
    }
    if u, err := url.Parse(a.URL); err == nil {
        return a.Acct + "@" + u.Host
    }
    return a.Acct
}

func short(s string, n int) string {
    s = strings.ReplaceAll(s, "\n", " ")
    if r := []rune(s); len(r) > n {
        return string(r[:n]) + "…"
    }
    return s
}

$ ./mastoscraper tag -pages 2 -out golang.csv golang
  mastodon.social  80 posts
  fosstodon.org    80 posts
  hachyderm.io     80 posts
240 posts fetched, 106 unique after de-duplicating by URI
    34 ★ resetreboot@masto.astropirad… Bueno, mastodones. Hoy por sorpresa y sin dar ni l…
    27 ★ golang@hachyderm.io          🥳 Go 1.27.3 and 1.26.10 are released!  🔏 Security:…
     7 ★ omkar_foss@mastodon.social   Dear #fediverse,  Introducing gracias, a #noai cli…
     6 ★ skybert@hachyderm.io         Lock down your website to a dedicated few browsers…
     5 ★ haitchfive@oldbytes.space    The Bathroom  Originally by Matthew Smith.  #zenpl…

240 posts fetched, 106 unique. More than half of what you download from a second and third server is stuff you already have. That's normal, and it's why de-duplicating by uri isn't optional. Also notice that the top post is in Spanish and not really about Go: hashtags are free text, so filter by language and your own keywords if you need precision.

Step 5: Trending hashtags per server

/api/v1/trends/tags returns each tag with a history array of daily uses and accounts. The first trap: history[0] is today, which is still being counted. My first run printed #tatort (0) and #FineArtAPlaceOrDestination (218) from today's partial numbers; switching to history[1] (yesterday, a complete day) gives comparable numbers. A tag can also trend today and still show 0 for yesterday, as you'll see below.

type TrendTag struct {
    Name    string `json:"name"`
    History []struct {
        Day      string `json:"day"`
        Uses     string `json:"uses"`
        Accounts string `json:"accounts"`
    } `json:"history"`
}

$ ./mastoscraper trends
mastodon.social  #SilentSunday (884), #audi (189), #OppositesAMovie (713), #FineArtAPlaceOrDestination (0), #tatort (72)
fosstodon.org    #SilentSunday (696), #Monsterdon (25), #Serbia (179), #CrispSandwichDay (51), #Tatort (31)
hachyderm.io     #silentsunday (717), #OppositesAMovie (641), #audi (113), #FineArtAPlaceOrDestination (0), #my9albums (45)

Trends are per server and moderated: admins review tags before they can trend. So "trending on mastodon.social" isn't "trending on Mastodon", and case varies (#SilentSunday vs #silentsunday). Lowercase tag names before you compare servers.

Step 6: CSV output and the CLI

func writeCSV(path string, posts []Status) error {
    f, err := os.Create(path)
    if err != nil {
        return err
    }
    defer f.Close()
    w := csv.NewWriter(f)
    w.Write([]string{"created_at", "author", "url", "favourites", "boosts", "replies", "language", "text"})
    for _, s := range posts {
        w.Write([]string{
            s.CreatedAt.Format(time.RFC3339), fullAcct(s.Account), s.URL,
            strconv.Itoa(s.FavouritesCount), strconv.Itoa(s.ReblogsCount), strconv.Itoa(s.RepliesCount),
            s.Language, plainText(s.Content),
        })
    }
    w.Flush()
    return w.Error()
}

func main() {
    if len(os.Args) < 2 {
        fmt.Println("usage: mastoscraper profile|posts|tag|trends ...")
        os.Exit(1)
    }
    cmd := os.Args[1]
    fs := flag.NewFlagSet(cmd, flag.ExitOnError)
    pages := fs.Int("pages", 3, "max pages per source")
    days := fs.Int("days", 30, "only posts from the last N days")
    servers := fs.String("servers", "mastodon.social,fosstodon.org,hachyderm.io", "comma-separated servers")
    out := fs.String("out", "", "write CSV here")
    fs.Parse(os.Args[2:])
    arg := fs.Arg(0)

    switch cmd {
    case "profile":
        a, server, err := lookup(arg)
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
        fmt.Printf("%s (%s) on %s\n  followers %d · following %d · posts %d · last post %s · bot=%v\n",
            a.DisplayName, a.Acct, server, a.FollowersCount, a.FollowingCount, a.StatusesCount, a.LastStatusAt, a.Bot)
    case "posts":
        since := time.Now().AddDate(0, 0, -*days)
        posts, err := userPosts(arg, *pages, since)
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
        }
        var fav, boost int
        for _, p := range posts {
            fav += p.FavouritesCount
            boost += p.ReblogsCount
        }
        fmt.Printf("%d original posts in the last %d days\n", len(posts), *days)
        if len(posts) > 0 {
            fmt.Printf("avg favourites %.1f · avg boosts %.1f\n", float64(fav)/float64(len(posts)), float64(boost)/float64(len(posts)))
            sort.Slice(posts, func(i, j int) bool { return posts[i].FavouritesCount > posts[j].FavouritesCount })
            for _, p := range posts[:min(3, len(posts))] {
                fmt.Printf("  %4d ★ %3d ⟳  %s\n", p.FavouritesCount, p.ReblogsCount, short(plainText(p.Content), 60))
            }
        }
        if *out != "" {
            writeCSV(*out, posts)
        }
    case "tag":
        seen, per := tagAcrossServers(strings.TrimPrefix(arg, "#"), strings.Split(*servers, ","), *pages)
        total := 0
        for srv, n := range per {
            fmt.Printf("  %-16s %d posts\n", srv, n)
            total += n
        }
        fmt.Printf("%d posts fetched, %d unique after de-duplicating by URI\n", total, len(seen))
        var list []Status
        for _, s := range seen {
            list = append(list, s)
        }
        sort.Slice(list, func(i, j int) bool { return list[i].FavouritesCount > list[j].FavouritesCount })
        for _, p := range list[:min(5, len(list))] {
            fmt.Printf("  %4d ★ %-28s %s\n", p.FavouritesCount, short(fullAcct(p.Account), 28), short(plainText(p.Content), 50))
        }
        if *out != "" {
            writeCSV(*out, list)
        }
    case "trends":
        for _, srv := range strings.Split(*servers, ",") {
            var tags []TrendTag
            if _, err := getJSON("https://"+srv+"/api/v1/trends/tags?limit=5", &tags); err != nil {
                fmt.Printf("%s: %v\n", srv, err)
                continue
            }
            names := []string{}
            for _, t := range tags {
                uses := "?"
                // History[0] is today (still counting); History[1] is yesterday, a full day.
                if len(t.History) > 1 {
                    uses = t.History[1].Uses
                }
                names = append(names, "#"+t.Name+" ("+uses+")")
            }
            fmt.Printf("%-16s %s\n", srv, strings.Join(names, ", "))
        }
    }
}

go build -o mastoscraper .
./mastoscraper posts -days 30 -out gargron.csv @Gargron@mastodon.social
./mastoscraper tag -servers mastodon.social,fosstodon.org -pages 2 -out golang.csv golang

Pitfalls, in one place

• The public timeline and status search need auth on mastodon.social (422 and empty results when logged out). Server admins decide this.
• Pagination lives in the Link header. If you only read the JSON body, you'll get one page and think you're done.
• id is per server, uri is global. De-duplicate on uri.
• acct is relative to the server you asked. Normalize to user@host.
• Remote copies lag. Ask the home server for profiles, and keep the highest counts when merging posts.
• Content is HTML. Strip it, and expect custom emoji shortcodes like :blobcat: in the text.
• Trends history[0] is today, still counting.
• Rate limits are per server and per IP (300 requests per 5 minutes unauthenticated on mastodon.social). Reading many servers spreads load, but be polite on each one.
• Not every fediverse account is Mastodon. Misskey, Pleroma, and others federate with Mastodon but have different APIs. The lookup above only works on Mastodon-compatible servers.

When you don't want to manage servers yourself

If you'd rather not juggle per-server rate limits, Link headers, and federated copies, I also run ScrapeAtlas*, which has Mastodon endpoints that wrap the same public API. For example, /v1/mastodon/hashtag takes a tag and an optional server, and /v1/mastodon/user/posts takes a handle with filter=original and a since date:

req, _ := http.NewRequest("GET", "https://api.scrapeatlas.com/v1/mastodon/user/posts?handle=Gargron@mastodon.social&filter=original&since=2026-09-26", nil)
req.Header.Set("x-api-key", os.Getenv("SCRAPEATLAS_API_KEY"))
// The JSON has "posts" (same fields as Mastodon's API), a "cursor" for the next page,
// and "coverage.warnings" that tell you when results are partial.

There are also /v1/mastodon/profile, /post, /post/replies, /post/interactions, /user/connections, /search, /trends, and /server. But honestly, for Mastodon the DIY route above is very workable. The API is open, documented, and stable.

Wrapping up

• Mastodon has a real public API. No HTML parsing, no headless browser.
• Think in servers. Every server has a partial view; read several and de-duplicate by uri.
• Go to the source for profiles and fresh counts.
• Read the headers: Link for pages, X-RateLimit-* for pacing.
• Respect opt-outs like noindex, and don't collect more than you need.

Which server-specific quirk have you hit while scraping the fediverse? Tell me in the comments and I'll add it to the list.

*I build it.