BitBit - A little compression library

Ever needed to compress a bunch of settings in a JavaScript object down to fit in a couple of bytes worth of data?  No?  Well that's okay I didn't either until recently.  While working with some embedded systems I needed a way to take simple number and boolean settings in an object, where I could store them in a readable format, and pack those down into under 2 bytes to be sent to and stored on a memory limited device.

Now, bit masks are useful if you're only dealing with multiple boolean values, but I needed to support integers as well, so I needed an easy way to say:  "Ok, bit 0 will be setting A, bits 1-6 will be setting B and bits 7-16 will be setting C."  So I created a little library called BitBit.

BitBit works by letting you create a new BitBit object with a schema that defines how a JavaScript object maps to certain bits.  For example, if you had a thermostat object with settings about a thermostat that you wanted to pack down into under 2 bytes, you could set up a new BitBit object like so:

What this is doing is creating a new BitBit object with a valid schema.  A schema must be an object with string keys that map to an array of 1 or 2 numbers.  A single number means it's a boolean and uses a single field.  2 numbers mean that its mapped to a number that uses the bits spanning the numbers given in the array (inclusive).  If you wanted to return a number that was just 0 or 1, you could use an array with 2 numbers that were the same.

Once you have built the schema, you can pack the settings down into an integer for storage.

As you can see above, once you pack the object, it will ignore anything not specified in the schema, which means you lose that data once it's unpacked.  A good way to use the unpacked data is to merge it back over your original object with lodash.merge.  This works well if the device you're sending it to and from can modify it.

In addition, you can use lodash accessor keys to get nested objects, array indices, etc.  There is an example of that on the Github Readme.

Well, that's about it, it's a small library I created since I needed something like that and didn't see anything out there that existed yet that satisfied those requirements.

If you like it feel free to contribute!  That's all for now, hopefully I'll have more time for these little side projects.

Card Creator V2 - A smoother workflow

I had a couple goals in mind when I set out to rebuild the Card Creator.  First, I wanted to be able to update a Google Spreadsheet document, and have the card creator read directly from there for building the cards, and second, I wanted to remove the Cairo dependency, because it wasn't working very well and I wanted to use Node4 which broke node-canvas.

Version 2 of the Card Creator does exactly that.  I hooked up the Card Creator into Google Sheets using the npm module: google-spreadsheet, and I swapped out node-canvas and Cairo for LibGD and node-gd.  What that entailed was a full rebuild of the card creation logic, but it was good to go back over it and update it and smooth out some of the minor issues.

In addition I redrew much of the art for the icons so that I could use non-pixel art instead, which scales better.  Granted, I am no artist, but I am doing all of the temporary art until I can find an artist to work with, if I decide to pursue this project seriously.

If you haven't read my previous post.  Card Creator is an application written in NodeJS that automates creating playing cards for a card game I'm working on.  It allows me to rapidly prototype and handles design, combining art assets, etc into print sheets for ease of production.

This new Card Creator program has improved my workflow significantly.  With the old program, making card changes required me to update the JSON files individually and change the values for each key/value pair in order to generate the cards.  Now I simply create a new row in my google document and run the program and it pulls in all the changes.  It's really nice for rapid prototyping.  I'm able to tweak values, update the descriptions, and then immediately run the program to generate the cards, and the sheetify script to create print sheets.

The script has a few points of interest you'll want to consider when using it.  First it's run by using the command: node creator-google-sheets.js which is the primary file.  (Make sure you're on branch version2).  The first thing you need to do to run it is include a google spreadsheet key to a google document.  In order to use the sheet, you'll need to publish it to the web, and then you'll have a link that has a long string in it.  That's your spreadsheet key.

I put my key in a private.js file which isn't included in the repository, so you can do the same, or you can simply replace the require('./private.js').google_sheets_key; on line 15 of the script with the string.  In addition if your spreadsheet has differently named headers for your spreadsheet, you'll need to update those in the mapping function (convertRow) on line 73.  That basically maps the row data to fields that the icon creator, background creator, etc are expecting.

The Card Creator works similar to before, it draws the background, the title, the description and the icons and then outputs it as a png file.  One thing to note, is that if you're working with node-gd, the documentation isn't fantastic, so it might take some trial and error to figure out how the functions work exactly.  For example, it took me a little bit to figure out which copy function worked for copying one image onto another.  Use copyResampled.  At least that's the only one that worked well for what I was doing.

I'm also rebranding the Card game I'm working on a bit, tossing some names around trying to find one I like.  I have a couple in mind, so we'll see if they stick in play testing.

Anyway feel free to check out the new Card Creator on my Github Page, if you're working with NodeJS linking up to google sheets documents is really quite simple and fantastic for automating work.  You can make the document read only (like I did) or allow programs to write to cells as well.  Nifty!

Cheers and thanks for reading!

Jason C / WakeskaterX

Introduction to Caching in JavaScript and NodeJS

Disclaimer:  This article is intended for new web developers who don't know much about caching and would like to have a basic idea.  It is not a comprehensive post, nor is it supposed to accurately reflect advanced caching systems.  All of the scripts referenced in this post can be downloaded from my GitHub Repository.

When I was first starting out in web development, I heard people talk a lot about this "cash" system that you could use in websites and your code, but I honestly didn't know too much about it.  I eventually figured out it was a way to store data so you could access it faster, but I still thought it was some complicated architecture or a special system, but I was really just over thinking it.

A Cache is simply a place in memory where you can store data that is accessed often.  In the context of JavaScript, this can simply be a global variable that you put the results of an AJAX call into.  Then you create your function to look in that first, and if something is there, return that, otherwise make the original call you wanted to.  There are a lot of libraries out there that will do that for you but we're going to build a very basic caching system, and walk through each of the parts.

I'm going to use NodeJS for this blog post, but this works just as well in the browser with an AJAX request or something similar.

First, I'll create a function that calls out to the github API and returns a value.  I'll walk through the function and then add some caching to show how it's done.  Below we have the basic script, which you can run with:

node nocache.js

For this tutorial, I am using the npm library request for ease of use and am making a request to the GitHub API to query my repositories for the data about them.  This is very similar to a request you would use in Express or another web framework and I have done things very similar for other projects.  One thing to note is that the request library lets you specify the options easily for the HTTP request and the GitHub API requires a User-Agent in the header, so that's why that's there.  If you don't GitHub will return an error and reject your request.

Next on line 15 I created the function to make the request.  For this tutorial I have a bunch of logs and start and end times to track the time in milliseconds the entire request takes.  So I set the start to +Date.now() which just converts Date.now to a number (in ms), and then logged the start time.  The data in the body comes as a JSON string, so I parsed that and logged the name first item since it's an array of information.

Finally, there are some logs to output the ending time, and the total time elapsed for the request.  On average I get roughly 400 ms on a good connection.  And if this is inside a web request, you don't want to be adding half a second to a second to every request, on top of everything else the server is doing.  To simulate this, the script nocache_multi.js has a few setTimeouts to repeatedly call the same function.  As you can see to the right when you run it, each time you get a similar response time.

This script is the perfect location to add caching because it's not changing the request parameters and we can pretty much expect that the response will be the same every time at least most of the time.  So instead of making the request every time the function is run, I'm going to add a storage object so that I can store the response and use that when the function is called again.

In the script below, you can see I've added a very basic cache named repo_cache on line 14, and added a result field to the object to store the data.  In a bit you'll see why I split it into a separate field but for now, but you can see how it's being used below.  On line 25 I added a check to see if we had any result data, if so, we simply log the results from that data and return, otherwise we continue with the original process.  In addition I split out the logging into a separate function so we can call it from each path.  The last change I made was that when I successfully get data, to store the parsed result in the repo_cache.result object.

When you run this function, you'll see that the first request takes some time, and then the next three are almost instantaneous.  Here's what my output looked like to the right.

As you can see the first request duration was 440 ms, and the rest were zero because we had the data in memory.

So I successfully "cached" the response and had a much better response time thereafter.  But we have a problem with this.  This kind of cache isn't very useful for a couple reasons.  First is that the data is going to get stale after a while and if the web server stays running for a while the data will be inaccurate, so there needs to be some kind of way to invalidate the cache, or turn it back off.  Well that's pretty easy to do, we just need to store a timestamp of when we generated the data for the cache as well as a timeout and then if the timestamp + the timeout is less than the current time, we make a new request and refresh the cache.

Here is a snippet of the script cache_advanced.js from the GitHub repo below:

The changes I made were adding a last_updated and timeout field to the cache object, as well as checking those 2 during the cache check, and updating the last_updated field after the request.

I got the following results to the right when I ran the script.  Since the timeout was set to 1 second and the time between requests was 1 second, I was able to cache the result for a single request, and then it was invalidated and refreshed and then accessed for the final function call.

So that's at a VERY basic level what caching is.  There are a lot of things you can do to improve it and learn more about caching like:  Using Function names and parameters to create a hash to store the results in and what not.  This function was just a quick and dirty way to create a cache for a script.  So hopefully if you're new to web development, that cleared up caching a bit and makes it a little easier to understand.

Feel free to download the Github Repository and grab all the scripts here:
https://github.com/WakeskaterX/Blog/tree/master/2016_04_Caching

JavaScript - Keeping your code DRY with Function Currying

If you are like me when I first heard about currying, you were probably thinking, well that sounds delicious, but what the heck is it?  Sadly, it is not the aromatic dish to the left, but it IS a super useful thing to know about in JavaScript, and really isn't that hard to understand and put into use once you get the hang of it.

There are a lot of really technical definitions and lots of great information out there about function currying, but if you haven't taken classes on algorithms or don't have a Math or CS degree, it can seem pretty confusing.  Function Currying is creating a function that returns another function with parameters set from the first function.  Ok, not so simple in words, but it's not that complex.  For an exact definition of it feel free to read up on it at Wikipedia, but since it's easier to just look at some code and see how it works, let's just take a look and see exactly what it is.

The Basic Example

To the right you can see a basic example that's often shown to demonstrate Functional Currying.  On line 3 you can see a very basic add function, it takes 2 inputs, and returns them added together.  That's simple, that makes sense, it works just like you'd expect it to.

On line 7 however, we have something that looks a bit different.  This is an example of function currying.  What is happening is that the function addCurried isn't returning a value, it's returning an entire function, and creating a scope, with Y set to the value used in the function.  On line 15 you can see an example of it being used to create a function called add_5 with the Y value in the function returned set to 5.  Then that function can be used, and it will work just like the add function, except that you only pass in the X value, because the Y is always set to 5.  You can also call it like addCurried(5)(3) to get 8 as well since addCurried returns a function which can then be called with the parameter (3).

If you don't know a ton about scope and closures in JavaScript and you're thinking:  How can this all be so?!  Go read You Don't Know JS - Scopes and Closures.  It will really take your understanding of JavaScript scopes to a new level (and it's free online).  Function currying isn't that scary at all, it's just a way to create a function with "preset values" so that you can reuse parts of a function.  This becomes extremely useful when building callback functions and helps keep your code DRY.

Practical Usage

There are TONS of uses for function currying, but one that I often employ is using function currying to create middleware or callbacks that can be modified for multiple functions.

For example, say you're using Express and you want to authenticate a user token against different scopes for different endpoints.  Middleware makes that super easy, but you could spend a lot of time with anonymous functions doing logic for each endpoint if you don't keep your code dry.

First let's take a look at a very basic section of code without function currying.  In this section of code below, you can see that I made 2 routes, and in each route, I'm checking a req.decoded_token.scopes value against a list of scopes required for this endpoint.  If the validation fails, the response redirects to another endpoint and doesn't allow the user to pass.

This works, but is messy and isn't very extensible, nor is it DRY at all.  An easy way to make this code look a lot nicer, is to use function currying, or to be more precise, partial function currying.  Since we have 2 values that change from endpoint to endpoint: the scopes to validate against, and the endpoint to redirect to, we can create a middleware creator to curry those values onto a common logic base.

With this second iteration of the code, you can see it looks a LOT nicer and certainly more DRY.  The function createMiddleware takes 2 parameters, our scopes we want to validate against, and the endpoint we want to redirect to, and returns a function which will be used as the middleware function for our application endpoints.  

Function Currying and Partial Currying can be extremely useful in JavaScript particularly when dealing with anonymous functions or callbacks, and I often employ it to build handlers and helpers that can create the anonymous functions I need while being flexible and letting me specify the parameters.

I hope these snippets of code were some examples of how you can use currying to keep your code DRY.  It is a really powerful skill to learn in JavaScript and having a good handle on how to use it properly will allow you to save a lot of time and effort.

NodeJS Basics - Object Patterns & Differences

Ever wondered when you should use a Singleton vs. using an object instantiated for just that script or even just for that call?  Not knowing the scope of the object can cause all sorts of issues in a NodeJS application if you don't carefully consider when it gets instantiated and what is allowed to access it.

I'll quickly walk through a couple scenarios which look very similar, but have a much different impact on your code.  This may be obvious to you if you're a long time NodeJS developer, but it's something that has tripped me up a few times in the past.

First, let's take a look at the Singleton model and how it works.

The Singleton Model

In the NodeJS Handbook, Fred K. Schott introduces the NodeJS Singleton:
"In most languages, sharing an object across your entire application can be a complex process. These Singletons are usually quite complex, and require some advanced modifications to get working. In Node, however, this type of object is the default. Every module that you require is shared across your application, so there’s no need for any special classes or extra code."

In NodeJS we can very quickly and easily create singletons, but it's important to be careful of when to use them, so you don't get yourself in trouble.

Let's take a look at a sample function, which will be our singleton model using a basic Object Oriented Approach.  There are other ways to write Singletons as well, but I'm writing the Singleton this way to focus on the small differences between Singletons and Instantiated Objects and how to not get tripped up by them.

Above we have a basic function: MyClass, with a single attribute: name, and single function on that class: askName.  Once it's added to the module.exports it is globally available for any of your scripts to require.  The important thing to note here is that you are creating a new instantiated object that gets put on the global modules list:  new MyClass("testy1"). Any script that accesses it references the instantiated object.  So in a second script, you could access it like so:

And if you had a third script running in the application you could run it there as well:

Now this is the important thing to remember about Singletons, and the reason you use this pattern:  If you change the name in the first script on the same NodeJS process, it will change it in the second script as well.  So the following test script which runs both scripts, will output "Hello, my name is Bob" twice:

If you swap the order of these scripts, it will output "testy1" as the name and then "Bob".  So just make sure you remember that when you modify values in a Singleton, that it will have that effect on anything else that is accessing it.

When to Use

The Singleton pattern is particularly useful whenever you need to have something store values across all scripts.  Think of it as being "super global" to the node process.  But that also means that you shouldn't use it if you need to change things between web requests or want to have the object do different things in different areas of your process.

With this pattern, you don't need to create the objects every time.  When you require it, you're accessing the created and instantiated object, so you can use it's methods and values right away.

The Object Oriented Model

So what if you don't want to use the Singleton, or it doesn't suit your needs?  Well instead of instantiating the object on the module.exports, you can simply pass the constructor back and let the calling script instantiate it.

This fits more along the lines of traditional object oriented programming.  Rather than one single global object, you allow the calling script to create each object as needed.

The above script is exactly the same as the Singleton model with the exception of line 10.  Instead of passing in a new MyClass with a set name, we simply pass in the constructor, the function MyClass.  Then in our scripts, when we require it, we must instantiate it, creating the objects with their own values.

There are 2 ways I normally go about instantiating an object from a module.  The first is to simply require the class, as seen on line 2, and then instantiate it (line 3) separately or when you need it.  Or, if you know that you only need to instantiate it once at the start of the script, say in the case where you have an error library or a debugging library that has settings specific to an individual script, you can instantiate the require, as seen on line 5.

When to Use

The Object Oriented pattern is useful any time you want something to be local to your script or function.  One thing to watch out for is the scope of the object and where and when you instantiate a new one.  If you're going to be using something for a certain web request, make sure you instantiate it during that web request within the function.

One recent 'doh' moment I had was when I instantiated my object globally, but was using it in web requests that came in.  So while Node was handling multiple requests, the logic the object was doing was being accessed by 2 different processes and was being corrupted during the request.  Moving the instantiation into function where I handled the request solved the issue.

Looking back it seems like a simple mistake, but it threw me for a loop for a while trying to figure out how the data was being corrupted.

There are plenty of other patterns as well, but I wanted to focus on the very small change between a Singleton and a Constructor based pattern that can make all the world of difference in your NodeJS code if you don't realize how it's different.

Hopefully reading this saves you a headache in the future!  Thanks and if you enjoyed this article, please follow me on Twitter & Subscribe to the blog!  Cheers!

Node Card Creator - Sheetify JS - Automating Print Sheet Creation

The NodeJS Card Creator is finally coming together as a tool I can use to streamline the development process.  If you haven't read the first blog post, I encourage you to go check out how I built the Card Creator and how you can set up your environment to do the same in my post: Card Creator - Automating Card Creation in NodeJS  You can also view the entire project here, at my GitHub account:  https://github.com/WakeskaterX/CardCreator

The next big piece -- a simple, but important piece of the automation process -- was finished just the other day: Sheetify.JS.  This Sheetify script takes all of the cards that are created in the cards folder and groups them into groups of 9, which fit onto a standard 8.5" x 11" sheet of paper.  It then draws the cards to these sheets and outputs them to a sheets folder to be ready for printing.  Having this change it allows me to rapidly prototype by changing values, running 2 scripts and then printing the desired print sheets.  There is of course, the cutting out of the cards, so perhaps I can build some kind of cutting stencil, but in the mean time this has vastly improved the prototype speed.

Here is an example of the output sheet:

In addition to Sheetify the other thing I updated in the script was the ability to specify card images for the artwork section.  I made a few shitty drawings in Paint.NET to test it, but it works very well.

Here is an example of a Fire Spell with the image added to the card:

And that's pretty much it!  The sheet automation has been super helpful, so the next step is finding an artist to work with me on the project and playtest, playtest, playtest to hammer down the mechanics!

Updates to the Card Creator & Work on Sorceror's Arena

I've got some updates to the Card Creator application out (which you can find on GitHub here:  https://github.com/WakeskaterX/CardCreator ) as well as some updates to a card game I'm working on tentively named Sorceror's Arena.

Now I'm no amazing designer, but I managed to improve the image quality a bit and have a slightly better design for the automated card creation.  Before I was doing a lot of the card creating by hand in Paint.NET and it certainly gets tedious as I make small changes and need to create demo print sheets.  Reducing that workload was the purpose of the Card Creator and it's finally starting to look better than the cards I've created by hand which is why I haven't been using it until now.

So here is a quick look at what my currently printed prototypes look like, what the old generated cards looked like, and the recent change I just made to the card creator.

Current Prototype

Old Generated Image

New Generated Image

Now it is a little hard to tell in the two generated images, but the Icon fidelity went up greatly, and the fonts look better too.  Before the generated images didn't have any kind of indicator for rarity, and so that was added as well to have some differentiation in the cards.

There is still a lot to do, I need images, background art, a decent background for the description box, and I need to create the JSON files for the Lightning and Water sets, all just to make a first decent prototype.  That said, I may still hold off on images until I can find an actual artist to work with.

Little by little I find time to work on it and polish the game, in addition to the play testing and adjusting card values so nothing is TOO overpowered.  The project is still very early so it'll be a long time before any kind of release ever happens.

Card Creator - Automating Card Creation in NodeJS

Card Creator is a little tool I've been working on for automating the creation of card graphics for a card game I'm working on (tentatively) called Sorcerer's Arena.  The purpose of the tool is to automate the combining of art and font assets from a set of JSON data into the image PNG assets needed to print the cards.

You can find the current version on GitHub and it works pretty well at the moment.  I need to find better images (not a phenomenal artist here) and I need to find some good fonts and backgrounds, but the program works.

Card Creator uses NodeJS and Node Canvas to render images in a canvas style way on the server.  I looked into a few alternatives like Graphics Magick  and Image Magick but neither were particularly suited to combining images easily in a layered format like the HTML5 Canvas is (and by extension Node Canvas, which uses Cairo).  By far, the installation and debugging of the setup of this project took the longest, so I'll go into the problems I came across and how I resolved them as well.

Setting It Up

First, this project was a pain in the arse to set up.  Cairo has many dependencies, all of which may or may not be working, and Node Canvas also has some issues as well with the latest versions.  First things first, you need Cairo.  My experience is with a Mac Book Pro, so it may differ if you have Windows or a Linux Distro.  

I ran into a few problems, namely I had issues with:

• Multiple Installations of Cairo (Brew vs MacPorts, etc)
• The Version of Cairo & Dependencies
• The Version of Node Canvas

I created the project in NodeJS with the latest version of Node Canvas, Async and put a small sample together using a few image and some text.  I set up my Git repository and then began getting things set up with Cairo which Node Canvas relies on.

I had installed Cairo for Mac a long time before, but never got things working with it.  That was installed with Brew, and when I started this project, I tried a new installation method using MacPorts.  Using MacPorts only caused more issues, because some things were referencing the Brew installation and dependencies and some were referencing the MacPorts, and everything just broke.

So I uninstalled everything from MacPorts, and uninstalled everything Cairo related from Brew and tried the whole re-installation from Brew, because Brew tends to be pretty solid.  I made sure that I had all the dependencies I needed for Cairo and Brew would tell me which I did not have linked correctly.  After the MacPorts debacle, I had to manually link a few of the Brew libraries to get them all set up. 

It ended up working, and I could generate images with Node Canvas but I still had issues.  Namely, I could not get using Fonts to work correctly and could not get them to render correctly either.  As I dove into this issue, part of it turned out to be a known issue with the latest version of Cairo on Mac OSX Machines, so ultimately to get around it, I used Brew to install all of the dependencies that Cairo required, I then uninstalled Cairo from Brew since the only version available on Brew was 1.14.0 which did not work correctly.  I then reinstalled a 1.12.X version directly from a the Cairo Releases FTP which cleared up a few issues with the font sizing and not rendering correctly.

However, I STILL had issues.  I could not load TTF Fonts into Node Canvas, which is something it is supposed to support.  So I started looking into this and found that it was a recent issue in Node Canvas.  So after some searching and debugging, I blew away the node modules folder and instead of using Node Canvas version 1.2.11 (the latest version), I hard set the version to 1.1.0.  And FINALLY everything worked.  I was able to load in True Type Font files and display them on the cards.  Using old versions of both Cairo and Node Canvas.  

Just a side note:  All this debugging and set up took me longer than it took me to make the current version of the application (~7 commits in at the time of writing this).

So now I had everything working, I could position text and images, use fonts, and build cards to my specifications.  Now I could build my application.

The Application

The application is fairly simple, if you've ever worked with the HTML5 Canvas before, it should feel very similar.  First, the main entry point for the app is the creator.js file.

Creator.js uses the following workflow to generate card graphics:

• First, load in the libraries, configs, and most importantly, the card sets in the /card_values directory.  This file is where all of the settings to generate each card are stored in JSON format.
• Next, Iterate through each card from the combined JSON files and perform the following:
• Create a canvas element with setting specified in a general config setting.
• Load all of the True Type Fonts in with the font loader library.
• Draw the Background of the card on the Canvas.
• Draw the Icons for the element types and stats, (and if this is a Sorcerer card, the miniature icons).
• Write the Description Text in the main box.
• Write the Title of the Card at the top.

And that's the entire flow.  Unlike some of my other blog posts I won't go too into the code but that's essentially how it all works.  

What this allows me to do is make small changes in the main configuration file, changes images, add images, change text or descriptions and update cards and quickly generate the entire set and images in seconds.  This allows me to try different things, modify entire sets of cards and not worry about having to manually create the cards in Paint.NET (which is what I was doing before).

As it is now, the quality of the cards isn't up to par with creating them as pixel art in Paint.NET, but I'm working to improve that and make the quality better.  Primarily I need to find a good way to create bordered fonts that aren't see through on the inside which most tend to be that I've found.  This application also paves the way for getting an artist on the project later and being able to quickly create cards as new art assets come in.

The end result of this project is that I can take something like this:

And turn it into this (below) with just a single command (node creator.js).

So far the application has been very handy for prototyping, and all of the code is open source so if you like making card games, feel free to extend the project and build on it for your own game!

I'll be building more on the application as well.  A few features I would like to add in the future are:

• Creating the JSON files directly from a CSV, Excel Document or Google Spreadsheets documents (since I normally work in Google Spreadsheets for building the cards).
• Better Fonts and possibly auto loading & mapping all fonts in the fonts directory.
• Images for the cards, currently none exist and none are attempted to be drawn in the creator code yet.
• Legit background (other than some grey boxes and a yellowish background color).

So that's my Card Creator application, if you haven't used Node Canvas I encourage you to check it out, but definitely pay attention when you're setting it up and potentially just use older versions to get it working correctly.  Once it's all set up, Node Canvas is pretty fun and easy to use for building server side imaging applications.


See Post #2 on the Card Creator:  http://blog.wakeskaterstudio.com/2015/11/updates-to-card-creator-work-on.html

Making a Real Time Multiplayer Online Game in NodeJS (Part 3)

In post 1 and post 2 of this series, I introduced and talked through the server side code for a project I'm working on called QuadPong: a real time game made in NodeJS for the purpose of practicing server side programming and synchronization.  You can view the source code on my GitHub.

Now we're going to go over the client side and talk about the code and libraries used to talk to the server and generate our client side rendering of the game.  I'll cover in this post:

• index.html & main.css - the HTML page the game is on and the CSS used
• socket_handler.js - the client side socket layer which handles talking to the server
• keypress.js & canvas_helper.js - two open source libraries/sections of code I used and why I picked them
• game.js - the main logic of the game

Main Page

First there is the index.html page which contains the structure for the game.  It's very basic: at the top I included a few important scripts such as JQuery, keypress.js and our CSS file.

The game div will contain the canvas once it is initialized.  It's where the game plays out.

The buttons div is something I added to not have to add much in the way of UI to the game.  Ideally these commands would be a part of the in game logic, but for this quick prototype they work as a hack.

The players table also shows a list of the in game players so you can see who is in the game.  It's not pretty, but it's functional: it shows the player number, whether or not they are human or a computer player, how much life they have remaining in the game, and which player is you.

In the CSS file there are only a few sections dealing mostly with centering and background base colors.  It's very basic and I didn't spend a lot of time on the design for this project.

Socket Handler

Socket handler -- as you can probably deduce -- handles the socket connections with the server.  At the top of the file is the initialization of socket.io.  I then pull in the player and game ids if they exist in localStorage as well as initialize a few other variables used in the game.  Some of these could probably be moved into the game.js file as an improvement.

Below the variables are a few events to set up the initial game state.  When the socket.io connection connects for the first time, it will emit the data saved in localStorage.  This allows the server to reply with any updated information.  The connect event also requests for the game list to show how many open games there are.  One potential improvement here would be to make the server return information about each game in the list to allow users to join a specific game or spectate games.  

Next is the update function.  When the client receives information from the server, it updates the game objects, sets the player data, updates the player list table and then calls update on the Canvas to render out the new data.  These variables and the updateCanvas function are declared in game.js which we'll look at in a bit.

There are also quite a few other basic game control events, which I'll list quickly out here:

• on reconnect - sets the game state and sets the game data appropriately
• on started_game - sets the game state to started and saves any adjusted game & player data
• on game_settings - sets the client settings to mirror the server settings
• on created_game - enter the waiting state for other players to join & save game id
• on joined_game - enter the waiting state and update the game settings with all of the player data of players in that game
• on added_computer - adds a computer player to the player list
• on added_player - adds a human player to the player list
• on game_list - updates the game list & updates the canvas - this is the reflection back from the server when the client emits request_game_list
• on sync_players - synchronizes the client side player list to the server list
• on no_games - an error saying that no available games can be joined when a player tries to join and can't
• on game_over - the event setting the end of game state and winner of the game

In addition to this long list of events to handle the game logic, I also have some helper functions to create games, join games, start games, update the player list, restart the client game (leaving the current room), and extract player_data.  Below you can see some of these functions.

Here is the save_data function which stores the information received from the server and stored in global objects into localStorage, add_player which adds a human player to the player table, and add_computer, which adds a computer player to the player table.  Again, it would have been wise to maintain a consistent function naming convention and if I refactored this portion I would use camelcase instead of underscore.

Create game sets the player number to 1, emits an event to the server to create a new game and set the caller as the main player, and adds a human player to the players table.  Join game emits an event to join a random open game with the player's stored id.

Start game emits an event to the server notifying it to start the specified game id and restart game is a client side function that resets the state of the client side, and leaves the game on the server side, so that the player can join a different game.

And at the bottom of our socket handler, is clear_player_list, which clears out the player table and extract_player_data which pulls player data from an object passed in. 

Another improvement I could make here is to pull out some of these helper functions into a separate library to keep this section of code dedicated to the socket handling only.  The player table stuff could really go in it's own file for maintenance as the code base grows.

Added Libraries and Code

Nearly all of the code in this project was written by me, but on the client side I used a few libraries I want to quickly note.  First I used JQuery, using it mostly for selectors in the socket handler script, but I also used two other more interesting sections of code.

KeyPress.JS

KeyPress is a library by dmauro that you can check out here:  https://dmauro.github.io/Keypress/.  I went with this library because it gave me a lot of flexibility with key events and let me bind and chain them in various ways.

In the section of code to the right, you can see a portion of the key registration.  KeyPress lets you register combinations for on key up and on key down events.  Since the player can be any one of the four players in the game, I wanted to map each direction in a specific way.  For instance, if the player is player number 4, on the left side of the game, pressing would move them up, but would translate to no action for players in slots 1 and 3.  

On the server, paddles only have 2 movement options, positive motion "R" (to the right facing the game) and negative motion "L".  So for player 4 would translate to a negative motion, whereas for player 2, translates to a positive motion.  I decided to handle this logic on the client side, for simplicities sake on the server, so when a key press event is generated, it calls out to a separate function, playerActionGenerator with a character representing the key press event.   corresponds to a "none" event or setting the player action to none.   

The playerActionGenerator function below handles these various directions.  First it checks if the state is in an active state, or a menu state which handles these events differently (for selecting menu items), and if it is in a game, it checks the player number and generates the appropriate action based on the direction.

The function playerAction then takes the direction passed in and builds the package to emit to the server to update the action on the server's copy of the player.

KeyPress ended up being pretty easy to use and it's pretty flexible on what it's capable of so I highly recommend using it.  I haven't even used most of the chainability of buttons and combinations that are possible.

In addition to KeyPress and JQuery, I used a snippet of code from a Stack Overflow post.  It sets the canvas DPI correctly so that text and objects render crisply instead of blurry and then creates and returns the Hi DPI canvas.  It works very well and I haven't had to modify it at all.

Game.JS

Finally let's look at the last portion of the client side game code in game.js.  This file also contains the KeyPress handling as well but as we've already gone over that I'll skip over those sections.

At the top of the file are a bunch of variables and declarations to set up the game state.  Most of what we'll look at here can be summed up as:  game.js draws on the canvas based on which state the game is in.  If the game is playing or in a "GAME" state, it will use the game objects which get updated by pushes from the server to draw the game objects, otherwise it will draw various menus and text.

First we have the onload function for the window, where we create the Hi DPI canvas, append it to the game div and register our KeyPress handler.

The entry point for most actions in this script is updateCanvas, seen below.  Any time actions from the server need to update the UI, updateCanvas is called.  Depending on the state of the game, one of the many draw functions will be called.  

Many of these functions simply draw text in various places in the game, but we'll look specifically at drawPlayers which is called in drawGame on line 109.  Now, the way I built the server coordinate system was a traditional Cartesian Coordinate System, with up being the positive Y direction and right being the positive X axis, with (0,0) at the center of the game.  Canvas, however uses the top left corner as (0,0) with down being the positive Y direction, so I made the convert function to simply convert the server system to the client one, including a size multiplier to scale the graphics.

The drawPlayers function uses convert to iterate through all four players and update the rectangle based on the bounding box coordinates received from the server.

Drawing the ball works very similarly and the rest of the draw functions simply are used for menus and text, which should be easy to understand and read through.  That's pretty much the entirety of the client side, it's fairly simple as it's mostly just getting positional updates from the server, sending player action updates to the server, and rendering the game out for the player to see.  Plus all of the objects in the game are basic geometric shapes.

So feel free to pull down the source code and play with it and extend the game or use it to make your own game.  You can view it all on my GitHub here: https://github.com/WakeskaterX/QuadPong

Thanks for reading and that's all I have!  The three part series on my little NodeJS side project is concluded.

Making a Real Time Multiplayer Online Game in NodeJS (Part 2)

In Part 1 of this blog series I walked through the idea and overall Architecture of QuadPong, a four player online pong game made in NodeJS and HTML5.  Now we're going to take a look at the server side code, walk through each portion of the code and review key portions of it.  We'll start with the entry point for the application, server.js and then jump all the way down to the deepest parts of the code, looking at the game object and physics classes.  We'll slowly work our way back up through the game loop to the socket layer, where we'll close out by reviewing how the server talks to the client.

Note:  This is quite a long post, so I do encourage you to get fresh cup of coffee or tea, get comfortable, and skip around to the interesting parts if you need to.  Also, you can view all of the source code on GitHub, which should be of help when going through this post.

Server.JS -- /server.js

If you've done any coding with NodeJS or Express, server.js is pretty self explanatory.  Only 30 lines long, server.js simply sets up our http server, connects to socket.io and routes our socket connections to our SocketHandler script.  Express also has a static function making it very easy to server our public folder with just a line of code.

The hosted site (http://quadpong.xyz) is served behind NGinx as well, but I chose to use Express to host the static files as it ends up being easier to configure for a small project like this one.

That's it for the entry point to the application.  We'll now go all the way down to our Math, Physics and Game Object libraries and take a look at what makes up the core of the application.

Libraries -- /app/lib

/app/lib/physics.js

First let's look at physics.js.  Primarily a Vector2 library (Math/Physics, whatever!) it also contains the Bounding Box class used by our Game Objects.  

Disclaimer:  I am not a mathematician and never took Linear Algebra in college, so my knowledge of Vector Mathematics is limited to what I've taught myself on Khan Academy and from what I've needed to know from working with 3D and 2D programming in the projects I've done.  I'm sure there are much more efficient formulas to do some of the things in this library.  However, it does do the things I need it to.

So, with that out of the way, I went with a system that used Vector2 as a coordinate, rather than describing the beginning and end of the vector.  All Vector2s are assumed to start at (0,0).  I kinda just made it up as I went, so its not a full fledged Vector2 Library and if you want to make it one, you'll need to do some work to it.  This is one area that could use a lot of refactoring and could be made much better.

So down all the way in /app/lib/physics.js we see on line 61, the Vector2 Class constructor.  Vector2s are basically x and y coordinates, with tons of functions to return the data needed.  

In the preceding lines you'll notice a bunch of private functions, like getDirectionFromVector and getVectorFromDirection, and you can take a look at those, but they're mainly just assisting functions for the main Vector2 functions.

 

There is one function before the constructor that I'd like to take note of: degreeAdjuster.  It's been a long time since I've done trigonometry, and I'm sure there's a better mathematical solution for this, but a simple way to adjust for the way Math functions round when given a value is to determine which quadrant you're in, and adjust the output degree accordingly so that you get an accurate direction from 0 to 360 degrees.  In other words, this makes sure that when I get a direction, it's always a degree between 0 and 360 which is what I need for the other functions.  It is a little hacky, but get's the job done.

In the lines following the Vector2 constructor, you'll find the functions tacked on to the prototype so that they can be called on a Vector2 object.  There are a lot of functions and they are primarily split into two types: those that mutate the object they are being called on, and those that return a new Vector2 object.  Because in JavaScript passing in an object is always by reference, I kept the idea that in general the functions should mutate the Vector2 object being used to call them.  However there are some functions that do not follow that convention, and the whole library could honestly use a significant refactor.  I also included a copy function which simply clones the object it's being called on. You'll also notice, most of the functions will return this so that they can easily be chained together.

On the left, are the normalize, copy, and getMagnitude functions.  Normalize is used to take a vector of any size and mutate it to its unit vector, having a magnitude of 1.  This function also uses a helper function roundToPlace, which simply takes a number and rounds it to a specific number of decimal places (in this case 2).  

Copy is used quite often, and is a very useful function for making a clone of a Vector2 to mutate and return for uses where you do not want to affect the original.  You'll see this used a lot in the classes file.

There is also the getMagnitude which is just a wrapper for our private getMagnitudeFromVector function, which given a vector, will calculate and return its magnitude.

Finally let's look at a few more conspicuous functions, rotateDegrees and reflect, which may not be quite as intuitive as some of the other functions.  rotateDegrees is used to rotate the current Vector2 around the origin a certain amount and then returns a new Vector2 based on that amount.  This is one of those functions that doesn't quite fit the pattern of some of the others and could use a rework.

The library also has a reflect function which given a normal vector, will reflect the current vector across that vector.  Think about a ball bouncing against a wall at an angle.  If the normal is the vector coming out from the wall, then it uses that vector to calculate the new vector of the bouncing ball after it hits the wall.  This function is primarily used when calculating the bounce of the ball against other game objects.

So enough about the Vector2 class, you can look on GitHub if you want to review the other functions, but they're fairly straightforward.  

Next in our Physics.JS file we find the BoundingBox class.  This is a simple rectangular bounding box used to determine our collisions in the game.  You can examine the constructor on line 251, with a generateBoundingBox function below it, which generates the x1, y1, x2 and y2 values from the position, width, and height.

I won't spend a ton of time talking about the bounding box as it is fairly simple.  There is an update function and the intersects function, which is used to determine if two bounding boxes intersect, seen here.

Other than that there are a few degree to radian conversion functions, helper functions and then it's packaged it all up in module.exports.

There are also couple other scripts of note here in our library: a math_extentions script which when required, will add a Clamp function to the Math class which simply clamps a value between a min and max, and a helper.js script which simply has a few guid creation functions.  Alright!  Now onto our classes.js script!

/app/lib/classes.js

Classes.js contains three classes:  GameObject, which is the main parent class for our hierarchy of classes, and the Ball and Paddle classes which are children from it.  The GameObject class contains a Vector2 position, a Vector2 velocity, a name and a BoundingBox.  It also has one function at the moment: intersects, which checks another GameObject's bounding box against its own using the function from our physics.js script.

Below is also the Ball class.  It inherits from GameObject (on lines 39 and 45) and has a few of it's own fields like colliding (boolean check if it's currently colliding with something), colliding_with (GameObject reference of the other object in a collision), and game (a reference to the game the object is in).

Above is also the updatePosition function, which given a delta time, updates the position based off of the velocity, updates the bounding_box and checks if it is still colliding with anything after it's moved a bit.  There are a few more functions: 

• bounce_x and bounce_y - swaps the x portion of the velocity and the y portion respectively
• reset - sets the ball position to (0,0)
• start_velocity - sets the initial velocity at the start of a point
• checkBounds - determines if the ball is out of the playing area and scores or bounces accordingly.

Note: One "to do" I have is when refactoring in the future, to be consistent with my snake (underscore) and camel cases.  I mix and match them for my function names and it's not the best looking.

There's also one more function we'll look at for the Ball class, bounce_against, which I feel deserves a look at.

Bounce against takes a game object that it is colliding with and checks if it is already colliding with something.  If not, it will collide with the game object and bounce against it, reflecting over the normal vector.  In addition, I wanted to add a portion of the paddles velocity to the ball, as well as the vector from the center of the paddle to the ball, to allow the player to have some control over the direction of the ball.

At the end of the function I added other_velocity, which is the velocity of the paddle the ball is colliding with, and the additional_velocity, which is a calculated velocity vector based on the position on the paddle where the ball was hit, and then I normalized the vector and set its magnitude to the original magnitude so the ball speed is not changed throughout the game.

Finally in our classes script is the Paddle class on line 161.  The Paddle class is the player in a game, it holds the player's remaining life in a game, the action a player or computer is taking, as well as a bunch of other variables that we'll look at in some of the functions.

The Action is the primary directive that a player on the client needs to send to the server.  Each Paddle has 3 action states:  Move Left ("L"), Move Right ("R"), and Nothing ("N").  These are updated from the client or the computer action function, seen below.

Players can be either human or computer, and if the player is a computer, it will be directed in the game loop to calculate its current action based on the ball position.  It's not complex AI by any means, but for the purposes of this game, it works quite well.

The other functions are there to simply calculate and update the position of the paddle based on the action, and update the width of the paddle to shrink it when a point is scored against that player.

In addition there are a few private helper functions used to calculate a starting ball direction, the positive_vector for a player which is used to determine which direction is "Right" in respect to the normal vector being forward, and a function to help generate the appropriate bounding box for a given player number.

That's it for the classes script.  This is a simple game, so there are only have 2 main classes: the ball and the player.  For a larger game, I'd recommend breaking these down into smaller files as this could easily get unweildly in a single file.

The Game Loop -- /app/game_loop.js

Now we're going to hop up a level to the Game Loop.  The game loop contains the bulk of the game flow and logic on the server.  Game extends the Event Emitter object, so that it can use those functions to emit updates to the client once per loop.

In /app/game_loop.js we can see the constructor and main logic for our game loop class.  On line 10, is the initial object for the game that houses all of the data and functions.  It sets up a few variables such as the game id, and some references to the players and ball object which are in the game.

The main game loop starts on line 28, in function gameLoop.  This function is run as the update function.  This game is built in NodeJS, so it's prudent to be careful of how much processing is chucked on the event loop, and while the game is doing all of the calculations synchronously, I didn't want to repeatedly call the gameLoop over and over without any break for the event loop to work on other things, because that would eat away all of the CPU I have available for the application and it wouldn't be able to run multiple games per process.

So instead of calling the loop again instantly I used setTimeout to queue another call of the function on the event loop stack a specific time later (generally 15 ms from the time the current loop was called which equates to roughly 60 fps, this time is set in the config).  The time between when this loop runs and the next call becomes our delta time as set on line 29.

The entirety of the game loop is pretty self describing within the function:

• Set our Delta Time and start time
• Move our Players
• Move the Ball
• Check for collisions between the players and the ball
• Check if the ball is out of bounds
• Set our end time
• Check if the game was won

That's the main game logic.  I also added a check to make sure that any loop over 30 ms gets logged so I can see if the server is getting slow as well, and then the function emits out the player and ball data to the socket handler, and checks if the game is over by checking the scores.

Finally it queues up another loop if the game is still active (status === 2) a few milliseconds later.

There is a wide range of both private and public functions on the game object, but we'll just take a look at a few of them, which are of note.

Move player paddles, move ball and check collisions are pretty self explanatory; they iterate over the stored objects, updating their positions, and then checking bounding boxes of the player against the bounding box of the ball.  

The end of game function I'll show since it emits the survivor out to the server side socket handler, which we'll look at in a bit.  This function checks the life of each player and if only one survivor remains, that player is emitted out to the client in a notification that the game is over.  The game status is then set to 3 indicating that the game is done, and can be cleaned up.

For public functions, some notable ones include, addPlayer and addComputer which, as you can imagine, add a player and a computer respectively to the game, and gameStart which puts the game into play and begins to run the game loop.  As you'll notice in the gameStart function, it sets the status to 2, which is used by the game loop to determine whether or not to continue to run.

Another function of note is the updatePlayerAction function, which when the game is started, will change the action of the player in the game.  This is the way the client sends data to the running game and how state among the players is changed.

And then the on point event will score against players and notify the client.

Thats pretty much all there is to the game loop.  It acts out a very basic game system, updating each actor in the scene, and calculating all of the simple physics that happen.  For a small game like this, the logic is fairly compact, so it don't use much processing power on each loop.

If you wanted to take this and do some complex pathfinding algorithms, it would be worth offloading that to a separate thread with a C or C++ library for NodeJS.  I haven't had the need to create my own library yet, but I know it's possible, since it's what makes the callback model work.

The last main section of code is the socket handler for the server: server_sockets.js.

Socket Handler -- /app/server_sockets.js

The socket handler script, server_sockets,js, is the main communication layer for the server side.  It primarily talks with the parallel script on the client side, socket_handler.js to send web socket communications back and forth via emitted events.

When a player connects, they hit handleSocket, which is triggered by socket.io when the client connects via our server.js connection event.  The client will then emit a "new_player" event, which is caught in the below function on our socket.

In this event, the handler checks if any games with the game_id passed in have a player with the player_id passed in, and if so, uses a bit of reconnection logic to determine which state the client game should be in, and what players are in that game.

The socket will then emit "reconnect" with the data, letting the client know it's reconnecting and it should take appropriate actions.  The player is added to the socket "room" for the game they are rejoining, so they can receive the appropriate events.  It'll also emit a "sync_players" event, to make sure the client is aware of all players in this game.  However if the player is not reconnecting, it'll simply create some credentials for the player, namely a player ID #, and send that back to the player and treat them as a new user.

There is also have a disconnect event, a leave game event, and a start game event which all are fairly self explanatory.  When emitted a game id for leave and start, the handler will either drop the player from the current game or try to start the waiting game that the player is in, and if it has enough players, start it up.

The next event below those three is our join game event.  In the current iteration of the game, a player cannot specify which game they would like to join.  They simply request to join an open game, and the server will find an empty game in the games list to pair them to.

Here the socket server searches for a game_id that has open slots, adds the player, and then emits back out a bunch of data.  The logic is as follows:

• First, have the player's socket join the game room broadcast
• Emit the game settings to the player
• Send back information about the game they are joining
• Emit to all players via io.emit (a broadcast) the new game list data
• Emit to the entire room for that game the new player joining
• If the game is full, emit back to the player trying to join that the game is full

And if there is a server error the handler notifies the player of the reason for the error.

There are a few more events which handle: updating actions, creating games, adding computer and human players, requesting games lists and so on.  The rest of these events are pretty straight forward.  It either modifies the game or updates the client back with information.

Other Scripts of Note

So that's the bulk of the back end portion of the application.  In addition to the main scripts, there is also have a configuration folder in /app/config which holds our game settings configurations, as well as some mocha tests in /app/test.  Currently there is just a test to make sure the physics library works as expected, but more should be added to ensure our scripts work properly.

I hope you enjoyed this lengthy post about the server side component to the game.  The next post should be much shorter and will talk about the client side, the canvas element, the socket handling, and some other small libraries I used to handle input and the like.

Feel free to follow me on Twitter, and check out the GitHub with all of the code for you to fork and build on!

Part 3