After a standard period of hibernation, I am awake and hungry, but first it's time to pay the dues so without wasting much of your time, let's jump right into it. This is a long due post in continuation with the first post on RESTful api with express-js and mongodb - Part 1

What's covered in this article?

As promised, I'm going to cover the following

Pre-requirements

As I mentioned, this blog is (as the title says), part two. If you haven't gone through RESTful api with express-js and mongodb - Part 1 then I'd strongly advise you to read it first. Other than that you'd want to get code that was produced as part of first blog from this github repository, also you'd need to switch to part-1 branch. In a nutshell, the following code with clone latest repo, checkout part-1 branch and install npm packages.

  1. git clone [email protected]:daveamit/mongo-crud-express-api.git && cd mongo-crud-express-api && git checkout part-1 && npm install

The Big Bang

So you must be wondering how it all began? Why npm start launches the app? Today we are going to take a peek under the hood, roughly, the following happens

Still not clear? Let's break it down. When you do npm start, npm goes and looks for scripts configuration area for start command, meaning, npm <script>. If you open up your package.json, you'll find a script section as shown below,

  1. {
  3. ...
  5. "scripts": {
  6. "start": "node ./bin/www"
  7. }
  9. ...
  11. }

What it means is, when someone issues start command just fire-up node ./bin/www command. So when you do npm start you are actually executing node ./bin/www.

Following is commented version of app.js

  1. // This line loads up the express package
  2. var express = require('express');
  3. // express() creates an application object
  4. // which we'll be using to configure middleware
  5. var app = express();
  7. // The path module is used to manipulate and calculate path
  8. var path = require('path');
  9. var favicon = require('serve-favicon');
  11. // Morgan is a logging module,
  12. // you guessed it right, Morgan as in Dexter Morgan
  13. var logger = require('morgan');
  14. // These parsers are help express parse information in cookies, url and body
  15. var cookieParser = require('cookie-parser');
  16. var bodyParser = require('body-parser');
  18. //This is where we load routes
  19. var routes = require('./routes/index');
  20. var users = require('./routes/users');
  22. // view engine setup, what we are saying to express is
  23. // the ./views directory contains all our views
  24. app.set('views', path.join(__dirname, 'views'));
  25. // Here are are stating that we are going to use handlebars
  26. // as the view engine, hence all view would be foo.hbs.
  27. app.set('view engine', 'hbs');
  29. // This will cause all the requests to be logged on the
  30. // console
  31. app.use(logger('dev'));
  32. // bodyParser.json() detects json content, and parses it and
  33. // loads it under req.body
  34. app.use(bodyParser.json());
  35. app.use(bodyParser.urlencoded({ extended: false }));
  36. app.use(cookieParser());
  38. // express.static is a middleware which handles all the
  39. // static content
  40. app.use(express.static(path.join(__dirname, 'public')));
  42. // and finally we mount the routes
  43. app.use('/', routes);
  44. // this just means, when a request comes for /users
  45. // just pass on the request to users middleware
  46. app.use('/users', users);
  48. // As I'll explain how middlewares work, for now just know that
  49. // If the execution comes that this point means that,
  50. // nigther the static content nor any of our routes
  51. // handled the incoming request, meaning, we were unable
  52. // to entertain it, meaning its a 404 and forward to error handler
  53. app.use(function(req, res, next) {
  54. var err = new Error('Not Found');
  55. err.status = 404;
  56. // This invokes next error handling middleware
  57. // down the pipline
  58. next(err);
  59. });
  61. // error handlers
  64. // if we are in development mode,
  65. // we will print full stack-trace
  66. if (app.get('env') === 'development') {
  67. app.use(function(err, req, res, next) {
  68. res.status(err.status || 500);
  69. // render causes our 'error.hbs' view to be
  70. // rendered with the details, message and error object
  71. res.render('error', {
  72. message: err.message,
  73. error: err
  74. });
  75. });
  76. }
  78. // if we are in production make sure that
  79. // no stack-traces leaked to user
  80. app.use(function(err, req, res, next) {
  81. res.status(err.status || 500);
  82. // render causes our 'error.hbs' view to be
  83. // rendered with the details
  84. res.render('error', {
  85. message: err.message,
  86. error: {}
  87. });
  88. });
  90. // and finally we export fully configured ready to run
  91. // application instance
  92. module.exports = app;

and /bin/www,

  1. #!/usr/bin/env node
  3. /**
  4. * Module dependencies.
  5. */
  7. // Load up the app.js in which we configure and
  8. // export our app
  9. var app = require('../app');
  10. var debug = require('debug')('mongo-crud-express-api:server');
  11. var http = require('http');
  13. /**
  14. * Get port from environment and store in Express.
  15. */
  17. var port = normalizePort(process.env.PORT || '3000');
  18. app.set('port', port);
  20. /**
  21. * Create HTTP server.
  22. */
  24. var server = http.createServer(app);
  26. /**
  27. * Listen on provided port, on all network interfaces.
  28. */
  30. server.listen(port);
  31. server.on('error', onError);
  32. server.on('listening', onListening);
  34. /**
  35. * Normalize a port into a number, string, or false.
  36. */
  38. function normalizePort(val) {
  39. var port = parseInt(val, 10);
  41. if (isNaN(port)) {
  42. // named pipe
  43. return val;
  44. }
  46. if (port >= 0) {
  47. // port number
  48. return port;
  49. }
  51. return false;
  52. }
  54. /**
  55. * Event listener for HTTP server "error" event.
  56. */
  58. function onError(error) {
  59. if (error.syscall !== 'listen') {
  60. throw error;
  61. }
  63. var bind = typeof port === 'string'
  64. ? 'Pipe ' + port
  65. : 'Port ' + port;
  67. // handle specific listen errors with friendly messages
  68. switch (error.code) {
  69. case 'EACCES':
  70. console.error(bind + ' requires elevated privileges');
  71. process.exit(1);
  72. break;
  73. case 'EADDRINUSE':
  74. console.error(bind + ' is already in use');
  75. process.exit(1);
  76. break;
  77. default:
  78. throw error;
  79. }
  80. }
  82. /**
  83. * Event listener for HTTP server "listening" event.
  84. */
  86. function onListening() {
  87. var addr = server.address();
  88. var bind = typeof addr === 'string'
  89. ? 'pipe ' + addr
  90. : 'port ' + addr.port;
  91. debug('Listening on ' + bind);
  92. }

TL;DR version

The npm start invoking node ./bin/www, /bin/www loads up the app.js, app.js loads up all the routes and configures the app, then /bin/www reads port information from the environment variables and finally listens to given port (3000 by default) for any http requests. Simple enough, hmm?

Middleware

That being said, let's understand one of the core concepts of express, the mighty middleware. Middleware is actually a very simple concept to understand, it is very similar to middleman, what middleman does is he take the request from one party and gets in touch with the other party, does what needs to be done and then provides a response back to the party which placed the request in the first place. So middleware are just simple functions with three arguments; request, response and next, they act as a middleman when express receives a request, express passes this request to each middleware in chain until it receives a response. In general, there are two kinds of middleware, first one being a normal middleware, and second one an error handling middleware.

Middleware concept is at the core of express.js and very important to understand, almost everything in express.js is a middleware from bodyparsers to loggers, from routes to error handlers. So it is very important to understand what are middlewares and how they work.

A middleware has following signature

  1. function aMiddleware(req, res, next){
  2. //req: an express request object
  3. //res: express response object, res.send, res.json,
  4. //res.render etc can be used to generate response
  6. /* the voodoo */
  7. }

Here, req is the express request object, the res is express response object and next is the Next Middleware in The Pipeline. You can either do your thing with res object like sending response or you can trigger the next middleware in the pipeline. > But you can't do both, i.e You can't write to the res and call next as well, if you do so, the response will be sent to the client as soon as res.json or similar function is called then next will be executed, keep in mind that the response has already been sent and now next can't send response again so if the next middleware tries to send a response then express would throw an exception, though client will never know about that because it already received a response with status 200. In a typical scenario, the next middleware might get similar error when it tries to render a response.

Error when response is set and still next is called.

A word of caution, if you do not write to res nor call next then that request will hang forever. And that is not what you'd want. Although this is pretty useful while implementing long polling.

Ok now, can you guess what the following middleware does?

  1. function iJustSitThereAndDoNothingMiddleware(req, res, next){
  2. next();
  3. }

Yes! you guessed it right, it just "Sits there and does nothing"! I call such middleware as connective middleware as they do not terminate the pipeline and connect the request to the next middleware. You must be wondering "why in the world would I want to write such a dumb middleware?", Well, you don't need to be so harsh, dumb is a very strong word for such a slick thing like this. For, eg, we can use such middleware to log stuff!

  1. function aLoggingMiddleware(req, res, next){
  2. //Assume log is a logging function and its pre-written
  4. //Log the request
  5. log(req);
  6. //Log the response
  7. log(res);
  9. //Call the next middleware
  10. next();
  11. }

Don't believe me? Look at following snippet from the good guys at morgan,

  1. function logger(req, res, next) {
  2. // request data
  3. req._startAt = #ff0000
  4. req._startTime = #ff0000
  5. req._remoteAddress = getip(req)
  7. // response data
  8. res._startAt = undefined
  9. res._startTime = undefined
  11. // record request start
  12. recordStartTime.call(req)
  14. function logRequest() {
  15. if (skip !== false && skip(req, res)) {
  16. debug('skip request')
  17. return
  18. }
  20. var line = formatLine(morgan, req, res)
  22. if (null == line) {
  23. debug('skip line')
  24. return
  25. }
  27. debug('log request')
  28. stream.write(line + '\n')
  29. };
  31. if (immediate) {
  32. // immediate log
  33. logRequest()
  34. } else {
  35. // record response start
  36. onHeaders(res, recordStartTime)
  38. // log when response finished
  39. onFinished(res, logRequest)
  40. }
  42. next();
  43. };
  44. }

They are Similar right? The above middleware is essentially,

  1. function logger(req, res, next){
  2. //blah blah blah
  3. //la la la
  4. //some more blah blah blah and la la la
  6. //Call the next middleware
  7. next();
  8. }

Ok, now can you guess what the following middleware does?

  1. function helloWorldMiddleware(req, res, next){
  2. res.json({ 'message': 'Hello world!' });
  3. }

Again, yes! you guessed it right! It just responds with {'message': 'Hello world!'}. I guess you are getting the hang of it. Now the following middleware,

  1. function sayHiIfRequestedMiddleware(req, res, next){
  2. if(req.query.sayhi){
  3. res.json({ 'message': 'Hi ' + req.query.sayhi + '!' });
  4. }
  5. else{
  6. next();
  7. }
  8. }

As you may have guessed if a request came like /foo?sayhi=bar and this middleware was attached in the pipeline, it would respond with {'message': 'Hi bar!'} and terminate the pipeline (not call next), if the request did not contain sayhi request parameter, then it would just call the next middleware. I hope it makes sense.

Using middleware

I think we should talk about app.use now, app.use comes in many flavours, but for now we are interested in two, first one being app.use(middleware) and second one being app.use(mountPath,middleware). mountPath can be a string, a regular-expression etc. So app.use('/foo', sayHiIfRequestedMiddleware) will cause sayHiIfRequestedMiddleware middleware to be executed when someone asks for /foo and not for /bar on the other hand, if we register it like app.use(sayHiIfRequestedMiddleware) then express will execute it each and every time irrespective of the request url (T&C Apply 😜).

The Order in which middleware is registered is very important, express executes each middleware as they appear, so the middleware registered first will be executed first (ofcourse if path and other criteria match). That is why express.static middleware was registered before users and routes were registered, to serve static content first and then dynamic content. Suppose there is a file ./public/hello and a middleware is defined as app.use('/hello',function(res, rep, next){ ... }) and a request came for GET /hello then the file ./public/hello would be served and the app.use('/hello'.. would never get called.

The other type of middleware is an error handling middleware, they are exactly like normal middleware and all the principles apply to them, but they are only triggered when something goes wrong. And to signal that something has gone terribly wrong, we need to call next with an argument like next(err). In a nutshell, next() calls next regular middleware and next(err) calls next error handling middleware in the pipeline. When an error handling middleware is called, all the regular middleware in the chain are skipped. Following is an error handler from our app.js file.

  1. app.use(function (req, res, next) {
  2. var err = new Error('Not Found');
  3. err.status = 404;
  4. // This invokes next error handling middleware
  5. // down the pipline
  6. next(err);
  7. });
  9. app.use(function(err, req, res, next) {
  10. res.status(err.status || 500);
  11. // render causes our 'error.hbs' view to be
  12. // rendered with the details
  13. res.render('error', {
  14. message: err.message,
  15. error: {}
  16. });
  17. });

In this case, the first middleware creates an error object with message 'not found' and status to 404 and invokes the next error handling middleware by next(err).

Let me demonstrate how middleware behaves with a simple app having a bunch of middleware,

  1. var express = require('express');
  2. var app = express();
  4. function aLoggingMiddleware(req, res, next){
  5. var debug = require('debug')('app:aLoggingMiddleware:');
  6. //Do some kind of logging
  7. debug('********************* --' + req.url + '-- ***********************')
  9. //Call the next middleware
  10. next();
  11. }
  12. function helloWorldMiddleware(req, res, next){
  13. var debug = require('debug')('app:aLoggingMiddleware:');
  15. debug('Responding with Hello World!');
  16. res.json({ 'message': 'Hello world!' });
  17. }
  19. function sayHiIfRequestedMiddleware(req, res, next){
  20. var debug = require('debug')('app:sayHiIfRequestedMiddleware:');
  21. if(req.query.sayhi){
  22. debug('Saying hi to ' + req.query.sayhi);
  23. res.json({ 'message': 'Hi ' + req.query.sayhi + '!' });
  24. }
  25. else{
  26. debug('Calling next middleware');
  27. next();
  28. }
  29. }
  32. app.use(aLoggingMiddleware);
  33. app.use(sayHiIfRequestedMiddleware);
  34. app.use('/hello',helloWorldMiddleware);
  36. app.use(function (req, res, next) {
  37. var debug = require('debug')('app:404 Middleware:');
  38. debug('Calling error middleware');
  39. var err = new Error('Not Found');
  40. err.status = 404;
  43. // This invokes next error handling middleware
  44. // down the pipline
  45. next(err);
  46. });
  48. app.use(function(err, req, res, next) {
  49. var debug = require('debug')('app:errorHandlingMiddleware:');
  50. debug('Rendering error page');
  51. res.status(err.status || 500);
  52. // render causes our 'error.hbs' view to be
  53. // rendered with the details
  54. res.render('error', {
  55. message: err.message,
  56. error: {}
  57. });
  58. });
  60. /*
  61. ...
  62. ...
  63. so on and so forth
  64. */

I have used debug instead of console.log for formatted, which will help to better understand the output. If you are not able to make sense of debug just think of it as a fancy console.log which prints label and calculate delay between calls.

This configuration will cause aLoggingMiddleware to be executed on every request and helloWorldMiddleware to be executed only when /hello resource is requested. So when a request comes with /hello then first aLoggingMiddleware and then helloWorldMiddleware will be executed, in that sequence.

So essentially it will produce following output (If you can't see what in there, you can open the image in new tab by right clicking and zoom).

Output from above code

Explanation
  1. request: /

    sayHi middleware calls next (sayhi param absent)
    404 middleware calls error middleware
    error middleware renders error page.
    (request terminates)

  2. request: /?sayhi=oddcode

    sayHi middleware handles and returns 'Hi oddcode!'
    (request terminates)

  3. request: /hello?sayhi=oddcode

    sayHi middleware handles and returns 'Hi oddcode!', This happens because "sayHi" middleware is registered before "helloWorld" middleware. (request terminates)

  4. request: /hello

    sayHi middleware calls next (sayhi param absent)
    helloWorld handles the request because it is mapped to `/hello` url
    (request terminates)

  5. request: /random-stuff

    sayHi middleware calls next (sayhi param absent)
    404 middleware calls error middleware
    error middleware renders error page.
    (request terminates)

  6. request: /random-stuff?sayhi=oddcode

    sayHi middleware handles and returns 'Hi oddcode!'
    (request terminates)

Routing

Now let's see how requesting /users executed the middleware in ./routes/users.js. You see, we have used router.get instead of app.use to register the middleware. The router.get is very similar to app.use, the only difference is that it will be invoked only for GET requests that match the mountPath in this case /. Now you guys must be stretching your heads and thinking that the router.get is set to listen to / then why does it respond to /users? I know what you are saying but there is one more piece to this puzzle. Notice that in our app.js we have mounted the users module at /users by app.use('/users', users), so request with any verb (GET, PUT or POST) to /users would trigger the users module, in now inside the users module, the route matching remaining url will be invoked, in our case the url is /users so the remaining url with respect to users mount point is /, so router.get('/' ...) got executed. If we add a route router.get('/hello' ...) in our users module then to invoke the route, we'd need to put a GET /users/hello, as GET /users will take the control up to users module (because of app.use('/users',users)) and now the relative url becomes /hello which matches with our routes.get('/hello'... middleware and invokes it.

I believe this must have helped you understand how middleware works. For further reading I'd advise to go to official express post.

I guess this is it for the second part, I know - this part was mostly theoretical and we didn't write much code, well middleware can get quite complicated and if you do not understand middleware then you will not be able to understand about 80% of code in a real life ready for production quality application written in express.js. In the next and final part I'll cover.

Hope you guys had fun! Feel free to comment below, See you in part three.

Cheers and Happy coding !!

Read more articles on Node.js: