Uploading large input files

This guide walks through how to use the Nextmv API to make runs with and retrieve results of large files. The process uses short-lived URLs to upload the large input file with a unique ID to submit in place of the input for the run.

Note that the process outlined on this page must be followed for submitting input files greater than 6 MB and retrieving output files greater than 6 MB.

You can use this alternative workflow to upload any size data (up to 500 MB) and retrieve any size output (up to 100 MB). However for files less than 6 MB in size, we recommend using the standard run processs (either the Nexmtv CLI run command or the HTTP API /runs endpoint).

Overview

The image below outlines the process of working with large input files within the Nextmv platform:

run_overview

How to upload a large input file

Follow the five steps below to upload your large input file, create a new run using this large input file and then retrieve the results of a large output file.

1. Request presigned URL

First you must request a presigned URL to upload your large input file to. This will return an upload ID and and upload URL.

Bash
Response

curl --location --request POST \
'https://api.cloud.nextmv.io/v1/applications/<app-id>/runs/uploadurl' \
--header 'Authorization: Bearer <API-KEY>'

This returned upload URL will only last for 15 minutes. If it expires, simply request a new one and take note of the new upload ID.

2. Upload large input file

Next, upload your input using the upload URL that was returned above. (You will use your upload ID in the next step to link your uploaded file as the input to use for the new run.)

Important! Note that the @ symbol prefixing the path to the input file MUST be included for your file to be processed as JSON. If you leave out the @ symbol the file will be processed as plain text and your run will fail.

Bash

curl --location --request PUT '<upload-url>' \
--header 'Content-Type: application/json' \
--data '@input.json'

An empty response means the upload was successful. Note that this may take a moment depending on the size of the input file.

3. Start run

Once the upload has completed, you can start a new run with the large input file using the upload ID from step 1. This upload ID is sent in the body of the run request in place of the input.

Bash
Response

curl --location 'https://api.cloud.nextmv.io/v1/applications/<app-id>/runs' \
--header 'Authorization: Bearer <API-KEY>' \
--header 'Content-Type: application/json' \
--data '{
    "upload_id": "<upload-id>"
}'

You should receive a run ID like normal as a response.

4. Get run result

To get the result of your run, you use the same endpoint as with a standard run but with the appended query parameter ?format=url. This will return a standard run result payload but instead of the run output included in the returned JSON, in its place will be a URL to download the output file.

Note that the ?format=url query parameter can be used regardless of your output size.

Bash
Response

curl --location 'https://api.cloud.nextmv.io/v1/applications/<app-id>/runs/<run-id>?format=url' \
--header 'Authorization: Bearer <API-KEY>'

5. Download output

The download URL can be accessed however you’d like: curl, a browser, etc. However, note that this download URL will only last for 15 minutes. If it expires, re-run the result endpoint from Step 4 to get a new output URL.

Bash

curl --location '<your-output-url>'

Uploading large input files

Making runs with large inputs using the Nextmv API