upload_tracker.py 如何避免素材上傳批次重跑時重複提交

I ran the same batch twice. Not on purpose, obviously. The upload script hit a network timeout halfway through pushing 40 files to Adobe Stock, I fixed what I thought was the issue, and re-ran the whole thing. Twenty-three files went up twice. Adobe's duplicate detection caught most of them, but the review queue took longer, and getting duplicates pulled is the kind of account noise you do not want when you are trying to build a clean submission history on three platforms at once.

That night I wrote upload_tracker.py. It is the least glamorous script in my pipeline and the one I would keep if I could only keep two. Here is what it does and why the problem it solves is worse than it looks.

Why double-posting is not a small mistake

Every stock platform treats duplicate submissions differently, but none of them treat them well. Adobe Stock rejects duplicates during review and repeated rejections of the same content can flag your account for quality issues. Vecteezy's review process similarly bounces content that already exists in your catalogue. Dreamstime flags duplicates on ingest. On all three, the CSV metadata has to match exactly what the platform expects anyway: Adobe wants its 5-column header, Vecteezy its 4-column header, Dreamstime its 15-column one. make_metadata.py handles that part. But no metadata script can save you from yourself when the same JPG appears in two upload sessions.

The real trap is the partial failure. Batch jobs fail in the middle. A 2 GB server, a slow connection, a platform hiccup, any of these can stop run one at file 17 of 40. If your recovery plan is "run it again", you have a bug you have not found yet.

The ledger, not the database

upload_tracker.py uses the Python standard library only. No install, no dependencies, no database server. It keeps a plain text ledger file and answers three questions:

python upload_tracker.py pending items.txt   # what is NOT done yet
python upload_tracker.py done "asset-001.jpg"
python upload_tracker.py status

The design rule I stuck to: the tracker records what you have completed, never what you have started. A file only gets marked done after the upload step confirms it. If a run dies mid-file, that file is still pending, and the next run picks it up exactly once. Marking things done optimistically is how the first version of my own tracker double-posted a test batch before I caught it. The fix was one line of ordering: confirm, then record.

This matters because the tracker is only as trustworthy as its weakest moment, and the weakest moment is always the crash window. Fail-then-record has no crash window that loses data. Record-then-fail has one that creates duplicates.

How it fits the batch flow

My folder layout is fixed: assets/ for raw output, output/ for submission-ready files plus the metadata CSVs, scripts/ for the tools. A batch run looks like this:

1. Generate assets into assets/ (one file per asset, gc.collect() between items, because 2 GB of RAM does not forgive parallel image jobs).
2. Run make_metadata.py against the folder for each platform.
3. Run upload_tracker.py pending to get the list of files not yet posted.
4. Upload only the pending list, platform by platform.
5. Mark each file done as its upload confirms.

Step 3 is the whole point. The re-run is now idempotent by construction. I have re-run that batch five times since, deliberately, and the pending list came back empty every time after the first complete run.

One honest limitation: the tracker does not know about the platforms themselves. It cannot query Adobe and ask "did this file actually land?" It trusts your upload step's confirmation. If your upload step lies, the tracker lies. Keep the upload step dumb and honest, and the ledger stays accurate.

Why plain text beats clever

I considered SQLite. I considered a JSON state file with locking. Both are fine. Plain text won because on a 1 vCPU box with 2 GB RAM, the simplest thing that cannot corrupt is the right thing. If the ledger gets mangled, I can read it, fix it by hand, and know exactly what state I am in. status prints a count and a list. That is the entire interface. Debugging a state store at 1 a.m. after a failed batch is not a problem I wanted to have twice.

The same philosophy runs through the rest of the pipeline. make_metadata.py writes the exact header each platform demands and nothing else, because all three reject a wrong header instantly. pack_product.py zips files and renders a 1280×720 cover and a 600×600 thumbnail with Pillow, the only image dependency in the kit. Everything runs on that same small server, one process at a time, peak RSS measured with ru_maxrss instead of guessing.

What I would tell past me

The failure mode was never the upload. It was assuming a re-run was safe. Any batch pipeline you run more than once needs a ledger that records completions, not attempts, and it needs to be the boringest possible file on disk. Build it before your first partial failure, not after, because the first partial failure is guaranteed and the duplicates it produces cost you review time and account reputation on every platform you sell through.

This tracker is one of four scripts in the pipeline starter kit, the same tools that packaged 39 products and 856 MB of output on a single-core, 2 GB server. If you are building sticker and asset products on a similar setup, the ai sticker factory playbook walks through the whole pipeline, including the generation side and the platform rules behind these CSVs. It is the documentation of what worked, not a promise of what will.

Get ai sticker factory playbook

Building digital products? Free guides & kits for digital product sellers.