The workflow, step by step

From a file on your disk to its restored version. The commands are for a shell with curl: Terminal on macOS (zsh) and Linux (bash), or Git Bash on Windows (it brings the openssl and split commands the steps use); every block works in bash and zsh as it is. Run them in one terminal window, in the folder that holds your file - the variables of step 0 are kept only in that window. The same in one Python program: The same in Python.

The steps at a glance
StepWhyEndpoint
1. Check your keyMakes sure the key works before you send gigabytes.GET /me
2. Open an uploadTells us the file's name and size; we answer how to cut it.POST /uploads
3. Send the chunksThe file in pieces, so a broken connection costs one piece, not the whole file.PUT /uploads/{upload_id}/chunks/{index}
4. Wait for the analysisThe file becomes a media; we learn its length, frame rate, borders and scenes.POST /uploads/{upload_id}/complete, GET /media/{media_id}
5. Price and startYou see the price (5a), then start the job (5b) - two separate blocks; the price is taken from your balance at the start.POST /quotes, POST /jobs
6. Follow the progressA job runs from seconds to hours; you ask how far it is.GET /jobs/{job_id}
7. Review keyframesColorization only: you approve the colours before they are carried to every frame.GET and POST /jobs/{job_id}/keyframes...
8. DownloadThe finished file.GET /jobs/{job_id}/result
9. Clean upFrees your storage; results are deleted after 60 days anyway.DELETE /jobs/{job_id}, DELETE /media/{media_id}

Step 0: Your settings

Set your key, secret and file once. FILE is the name of your video file in the current folder. The last line clears the idempotency keys of an earlier file (steps 2 and 5b make new ones): run this block again for every new file.

0. Settings
API=https://www.recuromedia.com/api/1.0
AUTH="YOUR_API_KEY:YOUR_API_SECRET"
FILE=A001_C003.mov
unset UPLOAD_KEY JOB_KEY

Step 1: Check your key

Why: if the key or secret is wrong, you learn it now and not after a long upload. The call: GET /me. The response: your account - user.email, plan.name, balance.amount (see the Me object). Wait: no. What can go wrong: the errors of the first call.

1. Who am I
curl -sS -u "$AUTH" "$API/me"

Step 2: Open an upload

Why: a large file cannot travel in one request; the upload is a container on our side that collects the pieces. The call: POST /uploads with the file's name and size in bytes (the first line measures it). The response: an Upload object. Note three fields: id (the upload's id, for the next steps), chunk_size (8388608 bytes = 8 MiB unless you asked for another size) and chunk_count (how many chunks to send). Wait: no. What can go wrong: MediaTypeUnsupported (a file type we do not take), FileTooLarge, TooManyUploads (10 open already), InsufficientStorage.

2. Open an upload
SIZE=$(wc -c < "$FILE" | tr -d ' ')
UPLOAD_KEY=${UPLOAD_KEY:-$(openssl rand -hex 16)}
curl -sS -u "$AUTH" "$API/uploads" -H "Content-Type: application/json" -H "Idempotency-Key: $UPLOAD_KEY" \
     -d "{\"file_name\": \"$FILE\", \"size\": $SIZE}"

(The second line makes an idempotency key once: a second run of the block keeps it. When the response gets lost - the connection breaks while you wait - paste the block again: the same key gives you the same upload, not a second one.)

Step 3: Send the file in chunks

Why: if anything breaks off - the connection, the laptop's sleep, the script - only the chunk on its way is lost: resume the upload with the chunks still missing, even days later. The call: PUT /uploads/{upload_id}/chunks/{index} once per chunk, numbered from 0, each with its SHA-256 so that a damaged chunk is noticed. Put the id of step 2 into the first line. split cuts the file into pieces of exactly 8388608 bytes named chunk_aaaa, chunk_aaab ... in the right order. The response: one line per chunk, {"index":0,"received_count":1}. Wait: no. What can go wrong: ChunkSizeMismatch (another chunk size than the upload's), ChunkDigestMismatch (damaged: send it again).

3. Send the chunks
UPLOAD=upl_5f0c2a9e41b7d3c8e6a1f094
split -a 4 -b 8388608 "$FILE" chunk_
INDEX=0
for PART in chunk_*; do
  DIGEST=$(openssl dgst -sha256 -binary "$PART" | base64)
  curl -sS -u "$AUTH" -X PUT "$API/uploads/$UPLOAD/chunks/$INDEX" -H "Content-Type: application/octet-stream" \
       -H "Repr-Digest: sha-256=:$DIGEST:" --data-binary "@$PART"
  echo
  INDEX=$((INDEX + 1))
done
rm chunk_*

(The loop sends one chunk at a time. Programs can send up to 4 at the same time - max_parallel - as restore.py does.)

Step 4: Complete the upload and wait for the analysis

Why: completing turns the chunks into one file - a media - and starts its analysis, which a job needs. The calls: POST /uploads/{upload_id}/complete, then GET /media/{media_id} until status is ready (or failed). The response: the completion gives media_id - put it into the line MEDIA=. The media then shows status queued, analyzing (with a stage), and finally ready with everything the analysis found (the Media object). Wait: the analysis of a 2-second clip took 4 seconds in our tests; a full reel takes minutes. The loop asks every 5 seconds. What can go wrong: UploadIncomplete (send the chunks it lists, complete again); status failed - the file is damaged, or does not match the sha256 you gave (error.code ChecksumMismatch): delete the media and upload again.

4. Complete and wait for the analysis
curl -sS -u "$AUTH" --data '' "$API/uploads/$UPLOAD/complete"; echo

MEDIA=med_4821
until curl -sS -u "$AUTH" "$API/media/$MEDIA" | grep -Eq '"status":"(ready|failed)"'; do sleep 5; done
curl -sS -u "$AUTH" "$API/media/$MEDIA"

Step 5: See the price, then start the job

Why: a job costs money: the quote shows how much before anything is taken. The calls: POST /quotes (5a), then - once you have read the price - POST /jobs (5b) with the same settings, in a block of its own. The filter keys are listed in Filters and GET /catalogue; a preset works too: {"preset_id": "pre_90"}. The response: the quote (Quote object) has price.amount and covers (true: your balance pays for it); the job (Job object) has its id - put it into JOB= in step 6 - and billing.state reserved: the price has been taken. Wait: no. What can go wrong: BalanceInsufficient (top up), JobExistsForMedia (the media has a job already), OptionConflict (settings that cannot be combined).

5a. See the price - free, nothing is started:

5a. See the price
SETTINGS='{"filters": ["noise", "medium_dust_and_scratches"]}'
curl -sS -u "$AUTH" "$API/quotes" -H "Content-Type: application/json" -d "{\"media_id\": \"$MEDIA\", \"settings\": $SETTINGS}"; echo

5b. Start the job when the price is right and covers is true - this takes the price from your balance. No response? Paste the block again: the same JOB_KEY gives you the same job, never a second one or a second charge.

5b. Start the job
JOB_KEY=${JOB_KEY:-$(openssl rand -hex 16)}
curl -sS -u "$AUTH" "$API/jobs" -H "Content-Type: application/json" -H "Idempotency-Key: $JOB_KEY" \
     -d "{\"media_id\": \"$MEDIA\", \"settings\": $SETTINGS}"; echo

Step 6: Follow the job's progress

Why: a job runs on our servers for seconds to hours, depending on the length, the frame size and the filters. The call: GET /jobs/{job_id}, again and again (polling). The response: status goes from queued to running (with stage and progress.percent) to completed; a colorization job stops at needs_action first (step 7). See Jobs for every status. Wait: the response's Retry-After header says 1 second while the job runs; the loop asks every 5 seconds, which is enough for a script. Never ask more often than once a second. What can go wrong: status failed with error.message - the charge comes back on its own; delete the job and try again.

6. Wait for the job
JOB=job_3150
until curl -sS -u "$AUTH" "$API/jobs/$JOB" | grep -Eq '"status":"(completed|failed|cancelled|needs_action)"'; do sleep 5; done
curl -sS -u "$AUTH" "$API/jobs/$JOB"

Step 7: Review the keyframes (colorization jobs only)

Why: a colorization job colorizes a few frames of every scene first and waits for you (status needs_action, needs_action keyframes), so that you can check the colours before they are carried to every frame. Skip this step for other jobs - or start colorization jobs with "keyframes": "auto" to approve them without review. The calls: GET /jobs/{job_id}/keyframes lists them with the addresses of their images (download them with -o, as the first example of Keyframes shows); optionally PUT to change the colour strength or leave keyframes out, or Modify to change a keyframe with a sentence; then POST .../approve. The response: the approval gives the job back with status running, stage propagating. Wait: after approving, go back to step 6 until the job is completed. What can go wrong: SceneNeedsKeyframe (every scene keeps one keyframe), ModifyInProgress (wait for a running change).

7a. List the keyframes and the addresses of their images:

7a. List the keyframes (colorization only)
curl -sS -u "$AUTH" "$API/jobs/$JOB/keyframes"; echo

7b. Approve them as they are, once you have looked at them - the job runs on:

7b. Approve the keyframes
curl -sS -u "$AUTH" --data '' "$API/jobs/$JOB/keyframes/approve"; echo

Step 8: Download the result

Why: the finished file is on our server until you fetch it (and for 60 days at most). The call: GET /jobs/{job_id}/result. --fail: when the API answers with an error instead of the file (for example ResultNotReady), curl saves nothing and prints curl: (22) The requested URL returned error: 409, and the end of the line (|| { ... }) deletes what an earlier run had saved - it is not your result. To read the API's own error message, run the address without --fail and -o restored.mov. -C - resumes an interrupted download: when the connection breaks, the part stays - run the same line again and it continues. Then compare the file's SHA-256 with result.sha256 of the job (step 6 printed it): both must be the same 64 characters (upper or lower case does not matter). On macOS write shasum -a 256 restored.mov; in Windows PowerShell (Get-FileHash restored.mov).Hash.ToLower(). Wait: as long as the download takes. What can go wrong: ResultNotReady (the job has not completed), NoResult (it failed or was cancelled).

8. Download
curl -sS --fail -u "$AUTH" -C - -o restored.mov "$API/jobs/$JOB/result" || { [ $? -eq 22 ] && rm -f restored.mov; }
sha256sum restored.mov

Step 9: Clean up

Why: media and results count against your storage; delete them once you have the file. The calls: DELETE /jobs/{job_id} and DELETE /media/{media_id}. The response: nothing (204 No Content). Wait: no. What can go wrong: nothing that matters: a NotFound means it is gone already.

9. Clean up
curl -sS -u "$AUTH" -X DELETE "$API/jobs/$JOB"
curl -sS -u "$AUTH" -X DELETE "$API/media/$MEDIA"

The same in Python

What you need: Python 3.8 or later (python.org; macOS and Linux often have it as python3, Windows as py). Nothing else: the program uses the standard library only. How to run it: save the code as restore.py next to your video file, put your key and secret into the line AUTH = ..., and run python3 restore.py A001_C003.mov (Windows: py restore.py A001_C003.mov).

What it does: steps 2 to 8 above - it sends 4 chunks at a time, polls with If-None-Match and Retry-After, shows the price and stops when the job would not start (covers false), resumes the download and checks the SHA-256. It can be stopped at any moment (Ctrl+C, a crash, a closed laptop): run it again with the same file and it carries on where it was - the missing chunks, the analysis, the job, the download - from what it keeps in A001_C003.mov.recuro.json next to the file. It restores with two filters (change SETTINGS); a colorization job stops at the keyframe review and tells you so. Two options: --clean-up also does step 9 after the download (deletes the job and the media on our side), and --retry starts a new job after a failed or cancelled one (it deletes that job first - the program tells you when).

restore.py
"""Recuro Media API v1: upload a file, restore it, download the result.

Usage:   python3 restore.py [--retry] [--clean-up] <file>        (Windows: py restore.py ...)
           --retry     after a failed or cancelled job: delete that job and start a new one of the same media
           --clean-up  after the download: delete the job and the media on our side (step 9)
Needs:   Python 3.8 or later, nothing else (standard library only).
Stopped (Ctrl+C, a crash, a closed laptop)? Run it again with the same file: it carries on where it was."""
import base64, hashlib, json, os, sys, time, urllib.error, urllib.request, uuid
from concurrent.futures import ThreadPoolExecutor

API = "https://www.recuromedia.com/api/1.0"
# Your API key and secret (Administrator Tools on the website), joined by a colon. Keep the b in front of the quotes.
AUTH = "Basic " + base64.b64encode(b"YOUR_API_KEY:YOUR_API_SECRET").decode()
SETTINGS = {"filters": ["noise", "medium_dust_and_scratches"]}   # what the job does: filter keys or {"preset_id": "pre_90"}
PARALLEL = 4                                    # chunks sent at the same time (at most max_parallel of the upload)
CLIENT = "restore-py/1.3"
STOP = []                                       # set by Ctrl+C: the chunks not sent yet are skipped


def call(method, path, body=None, data=None, headers=None):
    """One request. Returns (status, headers, JSON answer or None). An error answer raises RuntimeError with its code."""
    h = {"Authorization": AUTH, "Recuro-Client": CLIENT}
    if body is not None:                        # a JSON body
        data, h["Content-Type"] = json.dumps(body).encode(), "application/json"
    elif data is not None:                      # raw bytes (a chunk)
        h["Content-Type"] = "application/octet-stream"
    h.update(headers or {})
    request = urllib.request.Request(API + path, data=data, method=method, headers=h)
    try:
        with urllib.request.urlopen(request, timeout=300) as r:
            status, answer_headers, raw = r.status, r.headers, r.read()
    except urllib.error.HTTPError as e:         # 3xx/4xx/5xx arrive here; 304 too
        status, answer_headers, raw = e.code, e.headers, e.read()
    if status >= 400:                           # an error: {"code": ..., "detail": ..., "request_id": ...}
        try:
            problem = json.loads(raw)
        except ValueError:                      # not our JSON: the error page of a proxy or web server on the way
            raise RuntimeError("HTTP %d for %s %s: %s" % (status, method, path, raw[:200].decode("utf-8", "replace").strip()))
        raise RuntimeError("%s: %s (%s)" % (problem.get("code"), problem.get("detail"), problem.get("request_id")))
    return status, answer_headers, (json.loads(raw) if raw else None)


def wait(path, finished):
    """Polls a media or job until finished(item) is true. A 304 (nothing changed) is free; Retry-After says when to ask again."""
    etag = None
    while True:
        status, headers, item = call("GET", path, headers={"If-None-Match": etag} if etag else {})
        if status == 200:
            etag = headers.get("ETag")
            print(path, item["status"], item.get("stage") or "")
            if finished(item):
                return item
        time.sleep(max(1, int(headers.get("Retry-After", "5"))))   # never faster than once a second


class State:
    """What a run has reached, next to the file (<file>.recuro.json): the upload, the media, the job. Belongs to this file only."""
    def __init__(self, path):
        self.file, info = path + ".recuro.json", os.stat(path)
        self.stamp = [info.st_size, int(info.st_mtime)]
        try:
            self.data = json.load(open(self.file))
        except (OSError, ValueError):
            self.data = {}
        if self.data.get("stamp") != self.stamp:  # another file of the same name: start afresh
            self.data = {"stamp": self.stamp}

    def __getitem__(self, key):
        return self.data.get(key)

    def __setitem__(self, key, value):
        self.data[key] = value
        with open(self.file, "w") as f:
            json.dump(self.data, f)

    def forget(self, *keys):
        for key in keys:
            self.data.pop(key, None)
        self["stamp"] = self.stamp                # (saved)


def send_chunk(path, session, index):
    """Chunk <index>: chunk_size bytes from index * chunk_size (the last one: the rest), with its SHA-256 (Repr-Digest)."""
    if STOP:
        return
    with open(path, "rb") as f:
        f.seek(index * session["chunk_size"])
        chunk = f.read(session["chunk_size"])
    digest = base64.b64encode(hashlib.sha256(chunk).digest()).decode()
    call("PUT", "/uploads/%s/chunks/%d" % (session["id"], index), data=chunk, headers={"Repr-Digest": "sha-256=:%s:" % digest})
    print("chunk", index, "sent")


def upload(path, state):
    """Steps 2 to 4: open (or resume) the upload, send the missing chunks, complete. Returns the media id."""
    if state["media_id"]:
        return state["media_id"]                  # uploaded on an earlier run
    session = None
    if state["upload_id"]:
        try:
            _, _, session = call("GET", "/uploads/" + state["upload_id"])      # the earlier run's upload: what is still missing?
            print("resuming", session["id"] + ":", session["received_count"], "of", session["chunk_count"], "chunks are there")
        except RuntimeError as e:                 # NotFound / UploadExpired (7 days without a chunk): start again
            print("the earlier upload is gone (%s) - starting again" % e)
    if session is None:
        sha256 = hashlib.sha256()                 # the whole file's SHA-256: we check it after the last chunk
        with open(path, "rb") as f:
            for block in iter(lambda: f.read(1 << 20), b""):
                sha256.update(block)
        if not state["upload_key"]:
            state["upload_key"] = str(uuid.uuid4())   # the same key after a lost answer: the same upload, never a second one
        _, _, session = call("POST", "/uploads", {"file_name": os.path.basename(path), "size": os.path.getsize(path), "sha256": sha256.hexdigest()},
                             headers={"Idempotency-Key": state["upload_key"]})
        state["upload_id"] = session["id"]
    while session["missing"]:                   # (the list holds up to 1,000 chunks: asked again until nothing is missing)
        pool = ThreadPoolExecutor(max_workers=min(PARALLEL, session["max_parallel"]))
        try:
            list(pool.map(lambda index: send_chunk(path, session, index), session["missing"]))
        except KeyboardInterrupt:
            STOP.append(True)                     # the chunks on their way finish; the others are skipped
            raise
        finally:
            pool.shutdown()
        _, _, session = call("GET", "/uploads/" + session["id"])
    _, _, done = call("POST", "/uploads/%s/complete" % session["id"], data=b"")
    state["media_id"] = done["media_id"]
    return done["media_id"]


def download(job, target):
    """Step 8: the result, resumed where an earlier run stopped (Range), then checked against result.sha256."""
    result = job["result"]
    if not result["url"]:
        raise RuntimeError("the result has no download address: the file has been deleted")
    have = os.path.getsize(target) if os.path.exists(target) else 0
    if have > result["size"]:                    # not this result: download it again
        os.remove(target)
        have = 0
    headers = {"Authorization": AUTH, "Recuro-Client": CLIENT}
    if have:
        headers["Range"] = "bytes=%d-" % have
    if have != result["size"]:
        request = urllib.request.Request(result["url"], headers=headers)
        with urllib.request.urlopen(request, timeout=300) as r, open(target, "ab" if r.status == 206 else "wb") as out:
            for block in iter(lambda: r.read(1 << 20), b""):
                out.write(block)
    with open(target, "rb") as f:
        if result["sha256"] and hashlib.sha256(f.read()).hexdigest() != result["sha256"]:
            os.remove(target)
            raise RuntimeError("the download does not match the job's SHA-256 - it was deleted; run this again to download it again")


def main(args):
    files = [a for a in args if not a.startswith("--")]
    flags = set(args) - set(files)
    if len(files) != 1 or not flags <= {"--retry", "--clean-up"} or not os.path.isfile(files[0]):
        sys.exit(__doc__.split("\n\n", 1)[1])   # the usage lines
    path = files[0]
    state = State(path)
    media_id = upload(path, state)
    media = wait("/media/" + media_id, lambda m: m["status"] in ("ready", "failed"))
    if media["status"] == "failed":
        sys.exit("the analysis failed (%s): %s\nTo upload the file again, delete %s and run this again." % (media["error"]["code"], media["error"]["message"], state.file))
    if state["job_id"] and "--retry" in flags:
        _, _, old = call("GET", "/jobs/" + state["job_id"])
        if old["status"] in ("failed", "cancelled"):
            call("DELETE", "/jobs/" + old["id"])
            print("deleted", old["id"], "(" + old["status"] + ") - starting a new job")
            state.forget("job_id", "job_key")
    if not state["job_id"]:
        _, _, quote = call("POST", "/quotes", {"media_id": media_id, "settings": SETTINGS})   # step 5: the price first
        print("price", quote["price"]["amount"], quote["price"]["currency"], "- starts:", quote["covers"])
        if not quote["covers"]:
            sys.exit("the job would not start: %(code)s: %(detail)s" % quote["refusal"])
        if not state["job_key"]:
            state["job_key"] = str(uuid.uuid4())     # the same key after a lost answer: the same job, never a second one
        _, _, job = call("POST", "/jobs", {"media_id": media_id, "settings": SETTINGS}, headers={"Idempotency-Key": state["job_key"]})
        state["job_id"] = job["id"]
    job = wait("/jobs/" + state["job_id"], lambda j: j["status"] in ("completed", "failed", "cancelled", "needs_action"))   # step 6
    if job["status"] == "needs_action":
        sys.exit("the job waits for your keyframe review (step 7): approve the keyframes, then run this again")
    if job["status"] != "completed":
        sys.exit("the job %s ended as %s%s - its charge was given back.\nTo start a new job of the same media: python3 restore.py --retry %s"
                 % (job["id"], job["status"], (": " + job["error"]["message"]) if job["error"] else "", path))
    target = "restored - " + job["result"]["file_name"]
    download(job, target)
    print("saved", target)
    if "--clean-up" in flags:                     # step 9
        call("DELETE", "/jobs/" + job["id"])
        call("DELETE", "/media/" + media_id)
        print("deleted", job["id"], "and", media_id, "on our side")
    os.remove(state.file)


if __name__ == "__main__":
    try:
        main(sys.argv[1:])
    except KeyboardInterrupt:
        sys.exit("\nstopped - run the same command again to carry on where it was")
    except (RuntimeError, OSError) as e:          # an error answer, the network, the disk
        sys.exit("stopped: %s\nRun the same command again to carry on; to start this file afresh, delete its .recuro.json file." % e)