How to handle file upload with NodeJS and express? It’s quite easy and there is many tutorials online but almost most of them can’t handle large files or I would not recommend to deploy them in production.

Why? It’s because when using libraries such as multer disk storage for handling the files, it works in the way that on a server side the file is loaded into the server memory and after it’s fully loaded, you can save it or disk or do anything else with it.
(Note: multer is working on the stream api for version 2.0 which is currently in alpha )

But this behavior is quite bad when you want to handle bigger files, let’s say 50MB+ , during the entire time of upload the memory on the server is used. And now imagine 10 users and 1GB files … so 10 GB of ram? Unacceptable.

How to do it?

So we need to stream the file directly to the disk while we are getting the data.
We will use busboy a streaming parser for HTML form data for node.js.

More specifically we will use middleware connect-busboy so we can still use Express.

All code is commented and should be quite self-explanatory.

First let’s register the busboy with express:

// Import busboy
const busboy = require('connect-busboy');

const app = express(); // Initialize the express web server
app.use(busboy({
    highWaterMark: 2 * 1024 * 1024, // Set 2MiB buffer
})); // Insert the busboy middle-ware

Next handle the post request:

// Handle the upload post request
app.route('/upload').post((req, res, next) => {

    req.pipe(req.busboy); // Pipe it trough busboy

    req.busboy.on('file', (fieldname, file, filename) => {
        console.log(`Upload of '${filename}' started`);

        // Create a write stream of the new file
        const fstream = fs.createWriteStream(path.join(uploadPath, filename));
        // Pipe it trough
        file.pipe(fstream);

        // On finish of the upload
        fstream.on('close', () => {
            console.log(`Upload of '${filename}' finished`);
            res.redirect('back');
        });
    });
});

Let’s run some tests:

Let’s start up the server and upload ~1.2GB file.

From “simple” monitoring of the resources we can see that memory consumption is not significantly increased when handling this large file.

Full code:

This is fully working code which you can try to run.

You can find steps to install/run here https://github.com/petrvecera/node-file-upload

const express = require('express');         // Express Web Server
const busboy = require('connect-busboy');   // Middleware to handle the file upload https://github.com/mscdex/connect-busboy
const path = require('path');               // Used for manipulation with path
const fs = require('fs-extra');             // Classic fs

const app = express(); // Initialize the express web server
app.use(busboy({
    highWaterMark: 2 * 1024 * 1024, // Set 2MiB buffer
})); // Insert the busboy middle-ware

const uploadPath = path.join(__dirname, 'fu/'); // Register the upload path
fs.ensureDir(uploadPath); // Make sure that he upload path exits


/**
 * Create route /upload which handles the post request
 */
app.route('/upload').post((req, res, next) => {

    req.pipe(req.busboy); // Pipe it trough busboy

    req.busboy.on('file', (fieldname, file, filename) => {
        console.log(`Upload of '${filename}' started`);

        // Create a write stream of the new file
        const fstream = fs.createWriteStream(path.join(uploadPath, filename));
        // Pipe it trough
        file.pipe(fstream);

        // On finish of the upload
        fstream.on('close', () => {
            console.log(`Upload of '${filename}' finished`);
            res.redirect('back');
        });
    });
});


/**
 * Serve the basic index.html with upload form
 */
app.route('/').get((req, res) => {
    res.writeHead(200, {'Content-Type': 'text/html'});
    res.write('<form action="upload" method="post" enctype="multipart/form-data">');
    res.write('<input type="file" name="fileToUpload"><br>');
    res.write('<input type="submit">');
    res.write('</form>');
    return res.end();
});

const server = app.listen(3200, function () {
    console.log(`Listening on port ${server.address().port}`);
});

 


9 Comments

mohammacd · 14.11.2018 at 13:32

thank you for your great tutorial , i want to know how to can add one text input and send data with file to express

Norris · 5.3.2019 at 8:10

This code works on local, but when upload to Firebase cloud functions, it does not work.
Any solution or suggestion please!

    pagep · 6.3.2019 at 15:37

    I don’t think you should be using this solution on Firebase. This is mostly the back-end code, with FB I think you should be using some of their API for storing the files, you know with this solution you are writing to a disk which is something you can’t do on Firebase as far as I know. Cheers Petr

Gary · 11.4.2019 at 18:19

Great tutorial, thanks! I have one problem: If the upload is very large and takes more than a couple of minutes, the browser complains of inactivity timeout. Is ist possible to “detach” busboy somehow and return immediately with a message like “upload in progress” to the browser without closing the stream before it’s finished?

BaiMaoLi · 22.10.2019 at 17:41

Where is the simple monitoring tool for centos?

BaiMaoLi · 24.10.2019 at 0:18

I have tried as your post,
I’ve tried with 1GB file upload with express js, and connect busboy, but it was not lucky.
When file size is big, it gives 503 error.
Is busboy upload file with multi thread based on chunked file?

https://primarytech.com · 20.11.2019 at 21:40

Wonderful resourceful information. I like what i have actually
acquired here. You make your site content pleasant and easy to grasp.
a
I can not wait to read more from you. Bookmarked!

Leave a Reply

Your email address will not be published. Required fields are marked *