This guide walks through how to use the Nextmv API to make runs with and retrieve results of large files. The process uses short-lived URLs to upload the large input file with a unique ID to submit in place of the input for the run.
Note that the process outlined on this page must be followed for submitting input files greater than 6 MB and retrieving output files greater than 6 MB.
You can use this alternative workflow to upload any size data (up to 500 MB) and retrieve any size output (up to 100 MB). However for files less than 6 MB in size, we recommend using the standard run processs (either the Nexmtv CLI run
command or the HTTP API /runs
endpoint).
Overview
The image below outlines the process of working with large input files within the Nextmv platform:
How to upload a large input file
Follow the five steps below to upload your large input file, create a new run using this large input file and then retrieve the results of a large output file.
1. Request presigned URL
First you must request a presigned URL to upload your large input file to. This will return an upload ID and and upload URL.
This returned upload URL will only last for 15 minutes. If it expires, simply request a new one and take note of the new upload ID.
2. Upload large input file
Next, upload your input using the upload URL that was returned above. (You will use your upload ID in the next step to link your uploaded file as the input to use for the new run.)
Important! Note that the @
symbol prefixing the path to the input file MUST be included for your file to be processed as JSON. If you leave out the @
symbol the file will be processed as plain text and your run will fail.
An empty response means the upload was successful. Note that this may take a moment depending on the size of the input file.
3. Start run
Once the upload has completed, you can start a new run with the large input file using the upload ID from step 1. This upload ID is sent in the body of the run request in place of the input.
You should receive a run ID like normal as a response.
4. Get run result
To get the result of your run, you use the same endpoint as with a standard run but with the appended query parameter ?format=url
. This will return a standard run result payload but instead of the run output included in the returned JSON, in its place will be a URL to download the output file.
Note that the ?format=url
query parameter can be used regardless of your output size.
5. Download output
The download URL can be accessed however you’d like: curl
, a browser, etc. However, note that this download URL will only last for 15 minutes. If it expires, re-run the result endpoint from Step 4 to get a new output URL.