Upload large files with S3 multipart upload - basics

By Daniel Aniszkiewicz · 12 August, 2021

Introduction

In today’s world, more and more people have access to the internet and file sharing is growing exponentially. Files are taking up more and more space, especially photos and videos where the quality is getting better and better. More and more people both in large metropolitan areas and in the countryside have access to fiber-optic Internet. The connections are increasingly stable, which makes downloading and uploading files trouble-free.


It happens, however, that not every country currently has a modern infrastructure related to the Internet, or we are in areas where the Internet is not fast and stable, and we would like to send files without problems. Can we do something about it?

If you are using the S3 service from AWS, and you are having trouble uploading large files, you might be interested in multipart upload.


General content

Multipart upload S3 allows you to upload a file that is split into small parts and finally merged into one inside S3. This approach gives us many advantages. In the case of poor internet, or internet that interrupts frequently, it allows us to pause, resume, and re-embed those parts of files that were not uploaded due to internet problems.

In the traditional approach, where the whole file is uploaded at once, there is nothing worse than when, with 99% of the file upload progress, you lose connection to the internet, and the file is not uploaded properly.


It is best to open images in a new tab.


s3

Process:

s3

Phases:

s3

s3

s3

Limits:


s3

Summary

In this article, we learned the basics related to multipart upload. There are now ready-made libraries both on the backend side that enable multipart upload from S3 and on the frontend side ready-made drag-and-drop libraries for uploading files. In the next posts we will cover the practice.