How to handle file upload with NodeJS and express? It’s quite easy and there is many tutorials online but almost most of them can’t handle large files or I would not recommend to deploy them in production.
Why? It’s because when using libraries such as multer disk storage for handling the files, it works in the way that on a server side the file is loaded into the server memory and after it’s fully loaded, you can save it or disk or do anything else with it.
(Note: multer is working on the stream api for version 2.0 which is currently in alpha )
But this behavior is quite bad when you want to handle bigger files, let’s say 50MB+ , during the entire time of upload the memory on the server is used. And now imagine 10 users and 1GB files … so 10 GB of ram? Unacceptable.
How to do it?
So we need to stream the file directly to the disk while we are getting the data.
We will use busboy a streaming parser for HTML form data for node.js.
More specifically we will use middleware connect-busboy so we can still use Express.
All code is commented and should be quite self-explanatory.
First let’s register the busboy with express:
// Import busboy const busboy = require('connect-busboy'); const app = express(); // Initialize the express web server app.use(busboy({ highWaterMark: 2 * 1024 * 1024, // Set 2MiB buffer })); // Insert the busboy middle-ware
Next handle the post request:
// Handle the upload post request app.route('/upload').post((req, res, next) => { req.pipe(req.busboy); // Pipe it trough busboy req.busboy.on('file', (fieldname, file, filename) => { console.log(`Upload of '${filename}' started`); // Create a write stream of the new file const fstream = fs.createWriteStream(path.join(uploadPath, filename)); // Pipe it trough file.pipe(fstream); // On finish of the upload fstream.on('close', () => { console.log(`Upload of '${filename}' finished`); res.redirect('back'); }); }); });
Let’s run some tests:
Let’s start up the server and upload ~1.2GB file.
From “simple” monitoring of the resources we can see that memory consumption is not significantly increased when handling this large file.
Full code:
This is fully working code which you can try to run.
You can find steps to install/run here https://github.com/petrvecera/node-file-upload
const express = require('express'); // Express Web Server const busboy = require('connect-busboy'); // Middleware to handle the file upload https://github.com/mscdex/connect-busboy const path = require('path'); // Used for manipulation with path const fs = require('fs-extra'); // Classic fs const app = express(); // Initialize the express web server app.use(busboy({ highWaterMark: 2 * 1024 * 1024, // Set 2MiB buffer })); // Insert the busboy middle-ware const uploadPath = path.join(__dirname, 'fu/'); // Register the upload path fs.ensureDir(uploadPath); // Make sure that he upload path exits /** * Create route /upload which handles the post request */ app.route('/upload').post((req, res, next) => { req.pipe(req.busboy); // Pipe it trough busboy req.busboy.on('file', (fieldname, file, filename) => { console.log(`Upload of '${filename}' started`); // Create a write stream of the new file const fstream = fs.createWriteStream(path.join(uploadPath, filename)); // Pipe it trough file.pipe(fstream); // On finish of the upload fstream.on('close', () => { console.log(`Upload of '${filename}' finished`); res.redirect('back'); }); }); }); /** * Serve the basic index.html with upload form */ app.route('/').get((req, res) => { res.writeHead(200, {'Content-Type': 'text/html'}); res.write('<form action="upload" method="post" enctype="multipart/form-data">'); res.write('<input type="file" name="fileToUpload"><br>'); res.write('<input type="submit">'); res.write('</form>'); return res.end(); }); const server = app.listen(3200, function () { console.log(`Listening on port ${server.address().port}`); });
11 Comments
mohammacd · 14.11.2018 at 13:32
thank you for your great tutorial , i want to know how to can add one text input and send data with file to express
pagep · 21.11.2018 at 18:35
It’s simple. Check out the busboy example here https://www.npmjs.com/package/busboy#examples it describes the upload with more fields. You would just add `busboy.on(‘field’` to the handling code.
Norris · 5.3.2019 at 8:10
This code works on local, but when upload to Firebase cloud functions, it does not work.
Any solution or suggestion please!
pagep · 6.3.2019 at 15:37
I don’t think you should be using this solution on Firebase. This is mostly the back-end code, with FB I think you should be using some of their API for storing the files, you know with this solution you are writing to a disk which is something you can’t do on Firebase as far as I know. Cheers Petr
Gary · 11.4.2019 at 18:19
Great tutorial, thanks! I have one problem: If the upload is very large and takes more than a couple of minutes, the browser complains of inactivity timeout. Is ist possible to “detach” busboy somehow and return immediately with a message like “upload in progress” to the browser without closing the stream before it’s finished?
amir · 24.6.2019 at 10:04
check this tutorial on mideum
https://medium.com/@pilovm/multithreaded-file-uploading-with-javascript-dafabce34ccd
BaiMaoLi · 22.10.2019 at 17:41
Where is the simple monitoring tool for centos?
BaiMaoLi · 24.10.2019 at 0:18
I have tried as your post,
I’ve tried with 1GB file upload with express js, and connect busboy, but it was not lucky.
When file size is big, it gives 503 error.
Is busboy upload file with multi thread based on chunked file?
https://primarytech.com · 20.11.2019 at 21:40
Wonderful resourceful information. I like what i have actually
acquired here. You make your site content pleasant and easy to grasp.
a
I can not wait to read more from you. Bookmarked!
Soubhik Chatterjee · 6.9.2020 at 10:49
I am getting an EPIPE error after a couple of successful uploads, details of which is explained here https://stackoverflow.com/questions/63761781/epipe-error-when-uploading-large-file-using-nodejs
Can you please help?
Soubhik · 14.1.2021 at 10:41
I have answered my own question about the EPIPE error, anyone else struggling with the same issue can refer my answer here https://stackoverflow.com/questions/63761781/epipe-error-when-uploading-large-file-using-nodejs/65716570#65716570