How to handle file upload with NodeJS and express? It’s quite easy and there is many tutorials online but almost most of them can’t handle large files or I would not recommend to deploy them in production.

Why? It’s because when using libraries such as multer disk storage for handling the files, it works in the way that on a server side the file is loaded into the server memory and after it’s fully loaded, you can save it or disk or do anything else with it.
(Note: multer is working on the stream api for version 2.0 which is currently in alpha )

But this behavior is quite bad when you want to handle bigger files, let’s say 50MB+ , during the entire time of upload the memory on the server is used. And now imagine 10 users and 1GB files … so 10 GB of ram? Unacceptable.

How to do it?

So we need to stream the file directly to the disk while we are getting the data.
We will use busboy a streaming parser for HTML form data for node.js.

More specifically we will use middleware connect-busboy so we can still use Express.

All code is commented and should be quite self-explanatory.

First let’s register the busboy with express:

// Import busboy
const busboy = require('connect-busboy');

const app = express(); // Initialize the express web server
app.use(busboy({
    highWaterMark: 2 * 1024 * 1024, // Set 2MiB buffer
})); // Insert the busboy middle-ware

Next handle the post request:

// Handle the upload post request
app.route('/upload').post((req, res, next) => {

    req.pipe(req.busboy); // Pipe it trough busboy

    req.busboy.on('file', (fieldname, file, filename) => {
        console.log(`Upload of '${filename}' started`);

        // Create a write stream of the new file
        const fstream = fs.createWriteStream(path.join(uploadPath, filename));
        // Pipe it trough
        file.pipe(fstream);

        // On finish of the upload
        fstream.on('close', () => {
            console.log(`Upload of '${filename}' finished`);
            res.redirect('back');
        });
    });
});

Let’s run some tests:

Let’s start up the server and upload ~1.2GB file.

From “simple” monitoring of the resources we can see that memory consumption is not significantly increased when handling this large file.

Full code:

This is fully working code which you can try to run.

You can find steps to install/run here https://github.com/petrvecera/node-file-upload

const express = require('express');         // Express Web Server
const busboy = require('connect-busboy');   // Middleware to handle the file upload https://github.com/mscdex/connect-busboy
const path = require('path');               // Used for manipulation with path
const fs = require('fs-extra');             // Classic fs

const app = express(); // Initialize the express web server
app.use(busboy({
    highWaterMark: 2 * 1024 * 1024, // Set 2MiB buffer
})); // Insert the busboy middle-ware

const uploadPath = path.join(__dirname, 'fu/'); // Register the upload path
fs.ensureDir(uploadPath); // Make sure that he upload path exits


/**
 * Create route /upload which handles the post request
 */
app.route('/upload').post((req, res, next) => {

    req.pipe(req.busboy); // Pipe it trough busboy

    req.busboy.on('file', (fieldname, file, filename) => {
        console.log(`Upload of '${filename}' started`);

        // Create a write stream of the new file
        const fstream = fs.createWriteStream(path.join(uploadPath, filename));
        // Pipe it trough
        file.pipe(fstream);

        // On finish of the upload
        fstream.on('close', () => {
            console.log(`Upload of '${filename}' finished`);
            res.redirect('back');
        });
    });
});


/**
 * Serve the basic index.html with upload form
 */
app.route('/').get((req, res) => {
    res.writeHead(200, {'Content-Type': 'text/html'});
    res.write('<form action="upload" method="post" enctype="multipart/form-data">');
    res.write('<input type="file" name="fileToUpload"><br>');
    res.write('<input type="submit">');
    res.write('</form>');
    return res.end();
});

const server = app.listen(3200, function () {
    console.log(`Listening on port ${server.address().port}`);
});

 


6 Comments

mohammacd · 14.11.2018 at 13:32

thank you for your great tutorial , i want to know how to can add one text input and send data with file to express

Norris · 5.3.2019 at 8:10

This code works on local, but when upload to Firebase cloud functions, it does not work.
Any solution or suggestion please!

    pagep · 6.3.2019 at 15:37

    I don’t think you should be using this solution on Firebase. This is mostly the back-end code, with FB I think you should be using some of their API for storing the files, you know with this solution you are writing to a disk which is something you can’t do on Firebase as far as I know. Cheers Petr

Gary · 11.4.2019 at 18:19

Great tutorial, thanks! I have one problem: If the upload is very large and takes more than a couple of minutes, the browser complains of inactivity timeout. Is ist possible to “detach” busboy somehow and return immediately with a message like “upload in progress” to the browser without closing the stream before it’s finished?

Leave a Reply

Your email address will not be published. Required fields are marked *