BitBit - A little compression library

Ever needed to compress a bunch of settings in a JavaScript object down to fit in a couple of bytes worth of data?  No?  Well that's okay I didn't either until recently.  While working with some embedded systems I needed a way to take simple number and boolean settings in an object, where I could store them in a readable format, and pack those down into under 2 bytes to be sent to and stored on a memory limited device.

Now, bit masks are useful if you're only dealing with multiple boolean values, but I needed to support integers as well, so I needed an easy way to say:  "Ok, bit 0 will be setting A, bits 1-6 will be setting B and bits 7-16 will be setting C."  So I created a little library called BitBit.

BitBit works by letting you create a new BitBit object with a schema that defines how a JavaScript object maps to certain bits.  For example, if you had a thermostat object with settings about a thermostat that you wanted to pack down into under 2 bytes, you could set up a new BitBit object like so:

What this is doing is creating a new BitBit object with a valid schema.  A schema must be an object with string keys that map to an array of 1 or 2 numbers.  A single number means it's a boolean and uses a single field.  2 numbers mean that its mapped to a number that uses the bits spanning the numbers given in the array (inclusive).  If you wanted to return a number that was just 0 or 1, you could use an array with 2 numbers that were the same.

Once you have built the schema, you can pack the settings down into an integer for storage.

As you can see above, once you pack the object, it will ignore anything not specified in the schema, which means you lose that data once it's unpacked.  A good way to use the unpacked data is to merge it back over your original object with lodash.merge.  This works well if the device you're sending it to and from can modify it.

In addition, you can use lodash accessor keys to get nested objects, array indices, etc.  There is an example of that on the Github Readme.

Well, that's about it, it's a small library I created since I needed something like that and didn't see anything out there that existed yet that satisfied those requirements.

If you like it feel free to contribute!  That's all for now, hopefully I'll have more time for these little side projects.

Card Creator V2 - A smoother workflow

I had a couple goals in mind when I set out to rebuild the Card Creator.  First, I wanted to be able to update a Google Spreadsheet document, and have the card creator read directly from there for building the cards, and second, I wanted to remove the Cairo dependency, because it wasn't working very well and I wanted to use Node4 which broke node-canvas.

Version 2 of the Card Creator does exactly that.  I hooked up the Card Creator into Google Sheets using the npm module: google-spreadsheet, and I swapped out node-canvas and Cairo for LibGD and node-gd.  What that entailed was a full rebuild of the card creation logic, but it was good to go back over it and update it and smooth out some of the minor issues.

In addition I redrew much of the art for the icons so that I could use non-pixel art instead, which scales better.  Granted, I am no artist, but I am doing all of the temporary art until I can find an artist to work with, if I decide to pursue this project seriously.

If you haven't read my previous post.  Card Creator is an application written in NodeJS that automates creating playing cards for a card game I'm working on.  It allows me to rapidly prototype and handles design, combining art assets, etc into print sheets for ease of production.

This new Card Creator program has improved my workflow significantly.  With the old program, making card changes required me to update the JSON files individually and change the values for each key/value pair in order to generate the cards.  Now I simply create a new row in my google document and run the program and it pulls in all the changes.  It's really nice for rapid prototyping.  I'm able to tweak values, update the descriptions, and then immediately run the program to generate the cards, and the sheetify script to create print sheets.

The script has a few points of interest you'll want to consider when using it.  First it's run by using the command: node creator-google-sheets.js which is the primary file.  (Make sure you're on branch version2).  The first thing you need to do to run it is include a google spreadsheet key to a google document.  In order to use the sheet, you'll need to publish it to the web, and then you'll have a link that has a long string in it.  That's your spreadsheet key.

I put my key in a private.js file which isn't included in the repository, so you can do the same, or you can simply replace the require('./private.js').google_sheets_key; on line 15 of the script with the string.  In addition if your spreadsheet has differently named headers for your spreadsheet, you'll need to update those in the mapping function (convertRow) on line 73.  That basically maps the row data to fields that the icon creator, background creator, etc are expecting.

The Card Creator works similar to before, it draws the background, the title, the description and the icons and then outputs it as a png file.  One thing to note, is that if you're working with node-gd, the documentation isn't fantastic, so it might take some trial and error to figure out how the functions work exactly.  For example, it took me a little bit to figure out which copy function worked for copying one image onto another.  Use copyResampled.  At least that's the only one that worked well for what I was doing.

I'm also rebranding the Card game I'm working on a bit, tossing some names around trying to find one I like.  I have a couple in mind, so we'll see if they stick in play testing.

Anyway feel free to check out the new Card Creator on my Github Page, if you're working with NodeJS linking up to google sheets documents is really quite simple and fantastic for automating work.  You can make the document read only (like I did) or allow programs to write to cells as well.  Nifty!

Cheers and thanks for reading!

Jason C / WakeskaterX

Introduction to Caching in JavaScript and NodeJS

Disclaimer:  This article is intended for new web developers who don't know much about caching and would like to have a basic idea.  It is not a comprehensive post, nor is it supposed to accurately reflect advanced caching systems.  All of the scripts referenced in this post can be downloaded from my GitHub Repository.

When I was first starting out in web development, I heard people talk a lot about this "cash" system that you could use in websites and your code, but I honestly didn't know too much about it.  I eventually figured out it was a way to store data so you could access it faster, but I still thought it was some complicated architecture or a special system, but I was really just over thinking it.

A Cache is simply a place in memory where you can store data that is accessed often.  In the context of JavaScript, this can simply be a global variable that you put the results of an AJAX call into.  Then you create your function to look in that first, and if something is there, return that, otherwise make the original call you wanted to.  There are a lot of libraries out there that will do that for you but we're going to build a very basic caching system, and walk through each of the parts.

I'm going to use NodeJS for this blog post, but this works just as well in the browser with an AJAX request or something similar.

First, I'll create a function that calls out to the github API and returns a value.  I'll walk through the function and then add some caching to show how it's done.  Below we have the basic script, which you can run with:

node nocache.js

For this tutorial, I am using the npm library request for ease of use and am making a request to the GitHub API to query my repositories for the data about them.  This is very similar to a request you would use in Express or another web framework and I have done things very similar for other projects.  One thing to note is that the request library lets you specify the options easily for the HTTP request and the GitHub API requires a User-Agent in the header, so that's why that's there.  If you don't GitHub will return an error and reject your request.

Next on line 15 I created the function to make the request.  For this tutorial I have a bunch of logs and start and end times to track the time in milliseconds the entire request takes.  So I set the start to +Date.now() which just converts Date.now to a number (in ms), and then logged the start time.  The data in the body comes as a JSON string, so I parsed that and logged the name first item since it's an array of information.

Finally, there are some logs to output the ending time, and the total time elapsed for the request.  On average I get roughly 400 ms on a good connection.  And if this is inside a web request, you don't want to be adding half a second to a second to every request, on top of everything else the server is doing.  To simulate this, the script nocache_multi.js has a few setTimeouts to repeatedly call the same function.  As you can see to the right when you run it, each time you get a similar response time.

This script is the perfect location to add caching because it's not changing the request parameters and we can pretty much expect that the response will be the same every time at least most of the time.  So instead of making the request every time the function is run, I'm going to add a storage object so that I can store the response and use that when the function is called again.

In the script below, you can see I've added a very basic cache named repo_cache on line 14, and added a result field to the object to store the data.  In a bit you'll see why I split it into a separate field but for now, but you can see how it's being used below.  On line 25 I added a check to see if we had any result data, if so, we simply log the results from that data and return, otherwise we continue with the original process.  In addition I split out the logging into a separate function so we can call it from each path.  The last change I made was that when I successfully get data, to store the parsed result in the repo_cache.result object.

When you run this function, you'll see that the first request takes some time, and then the next three are almost instantaneous.  Here's what my output looked like to the right.

As you can see the first request duration was 440 ms, and the rest were zero because we had the data in memory.

So I successfully "cached" the response and had a much better response time thereafter.  But we have a problem with this.  This kind of cache isn't very useful for a couple reasons.  First is that the data is going to get stale after a while and if the web server stays running for a while the data will be inaccurate, so there needs to be some kind of way to invalidate the cache, or turn it back off.  Well that's pretty easy to do, we just need to store a timestamp of when we generated the data for the cache as well as a timeout and then if the timestamp + the timeout is less than the current time, we make a new request and refresh the cache.

Here is a snippet of the script cache_advanced.js from the GitHub repo below:

The changes I made were adding a last_updated and timeout field to the cache object, as well as checking those 2 during the cache check, and updating the last_updated field after the request.

I got the following results to the right when I ran the script.  Since the timeout was set to 1 second and the time between requests was 1 second, I was able to cache the result for a single request, and then it was invalidated and refreshed and then accessed for the final function call.

So that's at a VERY basic level what caching is.  There are a lot of things you can do to improve it and learn more about caching like:  Using Function names and parameters to create a hash to store the results in and what not.  This function was just a quick and dirty way to create a cache for a script.  So hopefully if you're new to web development, that cleared up caching a bit and makes it a little easier to understand.

Feel free to download the Github Repository and grab all the scripts here:
https://github.com/WakeskaterX/Blog/tree/master/2016_04_Caching

JavaScript - Keeping your code DRY with Function Currying

If you are like me when I first heard about currying, you were probably thinking, well that sounds delicious, but what the heck is it?  Sadly, it is not the aromatic dish to the left, but it IS a super useful thing to know about in JavaScript, and really isn't that hard to understand and put into use once you get the hang of it.

There are a lot of really technical definitions and lots of great information out there about function currying, but if you haven't taken classes on algorithms or don't have a Math or CS degree, it can seem pretty confusing.  Function Currying is creating a function that returns another function with parameters set from the first function.  Ok, not so simple in words, but it's not that complex.  For an exact definition of it feel free to read up on it at Wikipedia, but since it's easier to just look at some code and see how it works, let's just take a look and see exactly what it is.

The Basic Example

To the right you can see a basic example that's often shown to demonstrate Functional Currying.  On line 3 you can see a very basic add function, it takes 2 inputs, and returns them added together.  That's simple, that makes sense, it works just like you'd expect it to.

On line 7 however, we have something that looks a bit different.  This is an example of function currying.  What is happening is that the function addCurried isn't returning a value, it's returning an entire function, and creating a scope, with Y set to the value used in the function.  On line 15 you can see an example of it being used to create a function called add_5 with the Y value in the function returned set to 5.  Then that function can be used, and it will work just like the add function, except that you only pass in the X value, because the Y is always set to 5.  You can also call it like addCurried(5)(3) to get 8 as well since addCurried returns a function which can then be called with the parameter (3).

If you don't know a ton about scope and closures in JavaScript and you're thinking:  How can this all be so?!  Go read You Don't Know JS - Scopes and Closures.  It will really take your understanding of JavaScript scopes to a new level (and it's free online).  Function currying isn't that scary at all, it's just a way to create a function with "preset values" so that you can reuse parts of a function.  This becomes extremely useful when building callback functions and helps keep your code DRY.

Practical Usage

There are TONS of uses for function currying, but one that I often employ is using function currying to create middleware or callbacks that can be modified for multiple functions.

For example, say you're using Express and you want to authenticate a user token against different scopes for different endpoints.  Middleware makes that super easy, but you could spend a lot of time with anonymous functions doing logic for each endpoint if you don't keep your code dry.

First let's take a look at a very basic section of code without function currying.  In this section of code below, you can see that I made 2 routes, and in each route, I'm checking a req.decoded_token.scopes value against a list of scopes required for this endpoint.  If the validation fails, the response redirects to another endpoint and doesn't allow the user to pass.

This works, but is messy and isn't very extensible, nor is it DRY at all.  An easy way to make this code look a lot nicer, is to use function currying, or to be more precise, partial function currying.  Since we have 2 values that change from endpoint to endpoint: the scopes to validate against, and the endpoint to redirect to, we can create a middleware creator to curry those values onto a common logic base.

With this second iteration of the code, you can see it looks a LOT nicer and certainly more DRY.  The function createMiddleware takes 2 parameters, our scopes we want to validate against, and the endpoint we want to redirect to, and returns a function which will be used as the middleware function for our application endpoints.  

Function Currying and Partial Currying can be extremely useful in JavaScript particularly when dealing with anonymous functions or callbacks, and I often employ it to build handlers and helpers that can create the anonymous functions I need while being flexible and letting me specify the parameters.

I hope these snippets of code were some examples of how you can use currying to keep your code DRY.  It is a really powerful skill to learn in JavaScript and having a good handle on how to use it properly will allow you to save a lot of time and effort.

NodeJS Basics - Object Patterns & Differences

Ever wondered when you should use a Singleton vs. using an object instantiated for just that script or even just for that call?  Not knowing the scope of the object can cause all sorts of issues in a NodeJS application if you don't carefully consider when it gets instantiated and what is allowed to access it.

I'll quickly walk through a couple scenarios which look very similar, but have a much different impact on your code.  This may be obvious to you if you're a long time NodeJS developer, but it's something that has tripped me up a few times in the past.

First, let's take a look at the Singleton model and how it works.

The Singleton Model

In the NodeJS Handbook, Fred K. Schott introduces the NodeJS Singleton:
"In most languages, sharing an object across your entire application can be a complex process. These Singletons are usually quite complex, and require some advanced modifications to get working. In Node, however, this type of object is the default. Every module that you require is shared across your application, so there’s no need for any special classes or extra code."

In NodeJS we can very quickly and easily create singletons, but it's important to be careful of when to use them, so you don't get yourself in trouble.

Let's take a look at a sample function, which will be our singleton model using a basic Object Oriented Approach.  There are other ways to write Singletons as well, but I'm writing the Singleton this way to focus on the small differences between Singletons and Instantiated Objects and how to not get tripped up by them.

Above we have a basic function: MyClass, with a single attribute: name, and single function on that class: askName.  Once it's added to the module.exports it is globally available for any of your scripts to require.  The important thing to note here is that you are creating a new instantiated object that gets put on the global modules list:  new MyClass("testy1"). Any script that accesses it references the instantiated object.  So in a second script, you could access it like so:

And if you had a third script running in the application you could run it there as well:

Now this is the important thing to remember about Singletons, and the reason you use this pattern:  If you change the name in the first script on the same NodeJS process, it will change it in the second script as well.  So the following test script which runs both scripts, will output "Hello, my name is Bob" twice:

If you swap the order of these scripts, it will output "testy1" as the name and then "Bob".  So just make sure you remember that when you modify values in a Singleton, that it will have that effect on anything else that is accessing it.

When to Use

The Singleton pattern is particularly useful whenever you need to have something store values across all scripts.  Think of it as being "super global" to the node process.  But that also means that you shouldn't use it if you need to change things between web requests or want to have the object do different things in different areas of your process.

With this pattern, you don't need to create the objects every time.  When you require it, you're accessing the created and instantiated object, so you can use it's methods and values right away.

The Object Oriented Model

So what if you don't want to use the Singleton, or it doesn't suit your needs?  Well instead of instantiating the object on the module.exports, you can simply pass the constructor back and let the calling script instantiate it.

This fits more along the lines of traditional object oriented programming.  Rather than one single global object, you allow the calling script to create each object as needed.

The above script is exactly the same as the Singleton model with the exception of line 10.  Instead of passing in a new MyClass with a set name, we simply pass in the constructor, the function MyClass.  Then in our scripts, when we require it, we must instantiate it, creating the objects with their own values.

There are 2 ways I normally go about instantiating an object from a module.  The first is to simply require the class, as seen on line 2, and then instantiate it (line 3) separately or when you need it.  Or, if you know that you only need to instantiate it once at the start of the script, say in the case where you have an error library or a debugging library that has settings specific to an individual script, you can instantiate the require, as seen on line 5.

When to Use

The Object Oriented pattern is useful any time you want something to be local to your script or function.  One thing to watch out for is the scope of the object and where and when you instantiate a new one.  If you're going to be using something for a certain web request, make sure you instantiate it during that web request within the function.

One recent 'doh' moment I had was when I instantiated my object globally, but was using it in web requests that came in.  So while Node was handling multiple requests, the logic the object was doing was being accessed by 2 different processes and was being corrupted during the request.  Moving the instantiation into function where I handled the request solved the issue.

Looking back it seems like a simple mistake, but it threw me for a loop for a while trying to figure out how the data was being corrupted.

There are plenty of other patterns as well, but I wanted to focus on the very small change between a Singleton and a Constructor based pattern that can make all the world of difference in your NodeJS code if you don't realize how it's different.

Hopefully reading this saves you a headache in the future!  Thanks and if you enjoyed this article, please follow me on Twitter & Subscribe to the blog!  Cheers!

Node Card Creator - Sheetify JS - Automating Print Sheet Creation

The NodeJS Card Creator is finally coming together as a tool I can use to streamline the development process.  If you haven't read the first blog post, I encourage you to go check out how I built the Card Creator and how you can set up your environment to do the same in my post: Card Creator - Automating Card Creation in NodeJS  You can also view the entire project here, at my GitHub account:  https://github.com/WakeskaterX/CardCreator

The next big piece -- a simple, but important piece of the automation process -- was finished just the other day: Sheetify.JS.  This Sheetify script takes all of the cards that are created in the cards folder and groups them into groups of 9, which fit onto a standard 8.5" x 11" sheet of paper.  It then draws the cards to these sheets and outputs them to a sheets folder to be ready for printing.  Having this change it allows me to rapidly prototype by changing values, running 2 scripts and then printing the desired print sheets.  There is of course, the cutting out of the cards, so perhaps I can build some kind of cutting stencil, but in the mean time this has vastly improved the prototype speed.

Here is an example of the output sheet:

In addition to Sheetify the other thing I updated in the script was the ability to specify card images for the artwork section.  I made a few shitty drawings in Paint.NET to test it, but it works very well.

Here is an example of a Fire Spell with the image added to the card:

And that's pretty much it!  The sheet automation has been super helpful, so the next step is finding an artist to work with me on the project and playtest, playtest, playtest to hammer down the mechanics!

Updates to the Card Creator & Work on Sorceror's Arena

I've got some updates to the Card Creator application out (which you can find on GitHub here:  https://github.com/WakeskaterX/CardCreator ) as well as some updates to a card game I'm working on tentively named Sorceror's Arena.

Now I'm no amazing designer, but I managed to improve the image quality a bit and have a slightly better design for the automated card creation.  Before I was doing a lot of the card creating by hand in Paint.NET and it certainly gets tedious as I make small changes and need to create demo print sheets.  Reducing that workload was the purpose of the Card Creator and it's finally starting to look better than the cards I've created by hand which is why I haven't been using it until now.

So here is a quick look at what my currently printed prototypes look like, what the old generated cards looked like, and the recent change I just made to the card creator.

Current Prototype

Old Generated Image

New Generated Image

Now it is a little hard to tell in the two generated images, but the Icon fidelity went up greatly, and the fonts look better too.  Before the generated images didn't have any kind of indicator for rarity, and so that was added as well to have some differentiation in the cards.

There is still a lot to do, I need images, background art, a decent background for the description box, and I need to create the JSON files for the Lightning and Water sets, all just to make a first decent prototype.  That said, I may still hold off on images until I can find an actual artist to work with.

Little by little I find time to work on it and polish the game, in addition to the play testing and adjusting card values so nothing is TOO overpowered.  The project is still very early so it'll be a long time before any kind of release ever happens.

Card Creator - Automating Card Creation in NodeJS

Card Creator is a little tool I've been working on for automating the creation of card graphics for a card game I'm working on (tentatively) called Sorcerer's Arena.  The purpose of the tool is to automate the combining of art and font assets from a set of JSON data into the image PNG assets needed to print the cards.

You can find the current version on GitHub and it works pretty well at the moment.  I need to find better images (not a phenomenal artist here) and I need to find some good fonts and backgrounds, but the program works.

Card Creator uses NodeJS and Node Canvas to render images in a canvas style way on the server.  I looked into a few alternatives like Graphics Magick  and Image Magick but neither were particularly suited to combining images easily in a layered format like the HTML5 Canvas is (and by extension Node Canvas, which uses Cairo).  By far, the installation and debugging of the setup of this project took the longest, so I'll go into the problems I came across and how I resolved them as well.

Setting It Up

First, this project was a pain in the arse to set up.  Cairo has many dependencies, all of which may or may not be working, and Node Canvas also has some issues as well with the latest versions.  First things first, you need Cairo.  My experience is with a Mac Book Pro, so it may differ if you have Windows or a Linux Distro.  

I ran into a few problems, namely I had issues with:

• Multiple Installations of Cairo (Brew vs MacPorts, etc)
• The Version of Cairo & Dependencies
• The Version of Node Canvas

I created the project in NodeJS with the latest version of Node Canvas, Async and put a small sample together using a few image and some text.  I set up my Git repository and then began getting things set up with Cairo which Node Canvas relies on.

I had installed Cairo for Mac a long time before, but never got things working with it.  That was installed with Brew, and when I started this project, I tried a new installation method using MacPorts.  Using MacPorts only caused more issues, because some things were referencing the Brew installation and dependencies and some were referencing the MacPorts, and everything just broke.

So I uninstalled everything from MacPorts, and uninstalled everything Cairo related from Brew and tried the whole re-installation from Brew, because Brew tends to be pretty solid.  I made sure that I had all the dependencies I needed for Cairo and Brew would tell me which I did not have linked correctly.  After the MacPorts debacle, I had to manually link a few of the Brew libraries to get them all set up. 

It ended up working, and I could generate images with Node Canvas but I still had issues.  Namely, I could not get using Fonts to work correctly and could not get them to render correctly either.  As I dove into this issue, part of it turned out to be a known issue with the latest version of Cairo on Mac OSX Machines, so ultimately to get around it, I used Brew to install all of the dependencies that Cairo required, I then uninstalled Cairo from Brew since the only version available on Brew was 1.14.0 which did not work correctly.  I then reinstalled a 1.12.X version directly from a the Cairo Releases FTP which cleared up a few issues with the font sizing and not rendering correctly.

However, I STILL had issues.  I could not load TTF Fonts into Node Canvas, which is something it is supposed to support.  So I started looking into this and found that it was a recent issue in Node Canvas.  So after some searching and debugging, I blew away the node modules folder and instead of using Node Canvas version 1.2.11 (the latest version), I hard set the version to 1.1.0.  And FINALLY everything worked.  I was able to load in True Type Font files and display them on the cards.  Using old versions of both Cairo and Node Canvas.  

Just a side note:  All this debugging and set up took me longer than it took me to make the current version of the application (~7 commits in at the time of writing this).

So now I had everything working, I could position text and images, use fonts, and build cards to my specifications.  Now I could build my application.

The Application

The application is fairly simple, if you've ever worked with the HTML5 Canvas before, it should feel very similar.  First, the main entry point for the app is the creator.js file.

Creator.js uses the following workflow to generate card graphics:

• First, load in the libraries, configs, and most importantly, the card sets in the /card_values directory.  This file is where all of the settings to generate each card are stored in JSON format.
• Next, Iterate through each card from the combined JSON files and perform the following:
• Create a canvas element with setting specified in a general config setting.
• Load all of the True Type Fonts in with the font loader library.
• Draw the Background of the card on the Canvas.
• Draw the Icons for the element types and stats, (and if this is a Sorcerer card, the miniature icons).
• Write the Description Text in the main box.
• Write the Title of the Card at the top.

And that's the entire flow.  Unlike some of my other blog posts I won't go too into the code but that's essentially how it all works.  

What this allows me to do is make small changes in the main configuration file, changes images, add images, change text or descriptions and update cards and quickly generate the entire set and images in seconds.  This allows me to try different things, modify entire sets of cards and not worry about having to manually create the cards in Paint.NET (which is what I was doing before).

As it is now, the quality of the cards isn't up to par with creating them as pixel art in Paint.NET, but I'm working to improve that and make the quality better.  Primarily I need to find a good way to create bordered fonts that aren't see through on the inside which most tend to be that I've found.  This application also paves the way for getting an artist on the project later and being able to quickly create cards as new art assets come in.

The end result of this project is that I can take something like this:

And turn it into this (below) with just a single command (node creator.js).

So far the application has been very handy for prototyping, and all of the code is open source so if you like making card games, feel free to extend the project and build on it for your own game!

I'll be building more on the application as well.  A few features I would like to add in the future are:

• Creating the JSON files directly from a CSV, Excel Document or Google Spreadsheets documents (since I normally work in Google Spreadsheets for building the cards).
• Better Fonts and possibly auto loading & mapping all fonts in the fonts directory.
• Images for the cards, currently none exist and none are attempted to be drawn in the creator code yet.
• Legit background (other than some grey boxes and a yellowish background color).

So that's my Card Creator application, if you haven't used Node Canvas I encourage you to check it out, but definitely pay attention when you're setting it up and potentially just use older versions to get it working correctly.  Once it's all set up, Node Canvas is pretty fun and easy to use for building server side imaging applications.


See Post #2 on the Card Creator:  http://blog.wakeskaterstudio.com/2015/11/updates-to-card-creator-work-on.html

NodeJS - A Little Experiment in Load Testing and Clustering

Load Testing NodeJS on Multiple Cores

Check Out the Git Hub project HERE

Something I wanted to investigate was using Node JS on a multi core system to do CPU intensive applications under high load, so I created a project to do just that.

There are a few ways to use multiple cores in NodeJS, two of which are Cluster (part of the NodeJS API) and WebWorker Threads.

I will preface this by saying I am no NodeJS expert, this project was simply for learning on my part and there may be a better way to go about this with an NGinx set up, but I wanted to share my findings and maybe you'll find it cool too.

The Planning Stages

I started the project out with a simple NodeJS and Express server, which when called with a number in the query, would calculate and return the Fibonacci sequence for that number to simulate high CPU calculations.

My first thought was to use Web Worker Threads to spin up a thread to do work on, and just pass in the data and let it work on a background thread, but this caused issues under high load as too many threads were being spun up (I had no cap at the time) and at the start they were descoping and causing segmentation faults.

I ended up fixing the Seg Fault issue, but even so, they were burning through all of my VMs 2GB of memory and crashing.  

I attempted to create a Worker Pool function to control the amount of threads at any one time, but it quickly got quite complicated.  The idea was to use a queue and simply drop information off and process when ready, but with web requests and waiting on data, this as well got very complicated.

Then I stumbled across the Cluster section of the NodeJS API and found my solution.  With the Cluster API you can spin up multiple instances of your node server on the same port.  So I decided to test if this truly would improve my performance.

Load Testing

I installed loadtest in order to deliver high concurrency testing to my application and began to test the application.  On one branch of my git project I had the vanilla, single threaded NodeJS server which would generate a Fibonacci sequence, and on my other branch I had my project which generated 4 servers on a single port, which would use up the multiple cores.

This was a MUCH simpler solution to building a multi-threaded system within a single event loop and was extremely easy to set up.

So the load testing began.  I called out to my application like so:

loadtest -t 20 -c 32 http://localhost:3030/fib?num=25

I pinged my application using various concurrency values for 20 seconds testing periods.  I tested up from 1 concurrent connection up to 1000 connections.

While I don't believe all four cores were actually being used (as it was on a VM and my computer didn't crash) I did see quite an improvement using the multiple core approach (as one would expect).

The Results

The results from the load test are as follows.  Load test outputs the maximum response times for 50%, 90%, 95%, 99%, and the maximum response time as well as Requests Per Second, Mean response time, and Total Requests.

Computer Specs:  The VM was an Ubuntu 14 VM with 2 GB Ram, 4 processors.

Each request was made with a Fibonacci sequence of 25 for 20 seconds.

NodeJS Single Event Loop

Concurrency	Completed Requests	50%(ms)	90%(ms)	95%(ms)	99%(ms)	Max (ms)	Req / Sec	Mean Lat (ms)
64	8891	135	169	180	245	286	429	150
200	8110	336	1276	1358	3300	7364	405	480
1000	8178	1235	3726	7626	15726	16256	409	2510

NodeJS Multi-Core with Cluster

Concurrency	Completed Requests	50%(ms)	90%(ms)	95%(ms)	99%(ms)	Max (ms)	Req / Sec	Mean Lat (ms)
64	15009	70	155	193	327	560	750	80
200	14498	252	476	567	756	1062	596	330
1000	14923	1172	1954	2538	3618	7641	718	1350

As you can see the results from using the multi-core approach were much better than the single event loop, and using Cluster is super easy to do.  

You can fork the GitHub project here:  https://github.com/WakeskaterX/NodeThreading

The main fork is (well master too, but) hostedVM which has the standard deployment of nodeJS and express, and the multi-core approach is on the hostedVM_multicore branch.

Feel free to test it as well and let me know how your results are with better machines than my very low powered VM.

Cheers,

WakeskaterX

Converting Visitors to Register - adding an Unobtrusive Pop Up

So after about a month or so of having my e-mail login active on my web page, I have exactly zero sign ups.  This isn't unexpected so I'm working on drawing more attention to the login and register buttons at the top.

A lot of websites see great results from using a full page popup when a new visitor visits the site, but generally when I see these I immediately leave the page, and it's fairly infuriating.

So my goals with the most recent pushes to the website were:

1.  To have a pop up that notifies new users about the registration button, but have it be off to the side and non obtrusive when viewing the website and easily closed.

2.  To make the registration process a bit more intuitive and flow a little easier.

For the first goal, I created a little speech bubble pop up that will show only if a user does not have login information saved on their computer, and when it pops up, it will only show for about ten seconds, and won't show back up for 24 hours.  I might extend this to 48 hours / a week, but I don't think users are visiting my site more than once within any length of time.

Here's an example of the pop up:

I wanted it to be mostly out of the way, and while it does cover up some buttons on the UI, it has a clear close button that closes the box immediately.

The second thing on my list was to make the registration process easier.  Before, the login and registration buttons were tied together, meaning you had to click them both and that would pop up the login prompt, then you had to click the register button, and then register.

These buttons have now been split out into a separate login and register button so that users can choose accordingly and not have to go through an extra click.

Just a few small UI updates that I've been meaning to do to convert more visitors into registered guests.  I'll post an update later on the results of the slight UI changes.


Cheers,
Jason C.

Creating a Rotating Gallery - Game Page Links for new Website

So the games page was supposed to be a simple panel of images that linked to pages for each game I have created and/or worked on.  But I thought to myself, "That's Boring!" so I created my own rotating gallery page viewer that flips through all the options as you click on them.

The rotating gallery uses JQuery, Javascript and CSS3, so knowing a bit about those will help you understand what's I'm doing.

First, here are a couple screen shots of the gallery:

When you click on a panel to each side, it rotates through the selection to that panel:

It also uses CSS transitions so it has a nice easing effect as well.

For each panel, they will link to a page specifically for that game.  Right now you enter via clicking the button at the bottom but I might change the coding so that when you click on the main picture, if it is active you will go to that page.

The code is a bit long so I won't go into too much detail, but I'll highlight a few of the places of note in the script and CSS.

The Main Page:

So the goal was to just add divs as I needed and to have the JavaScript file manipulate and add the divs to a list and control them that way.

I wanted the main page to be easy to add things to and all the work to be hidden in JS.

As you can see it's pretty intuitive and makes a lot of sense what's going on.  You have 5 game pages, each with a title and an image.  I kept the actual links in the JavaScript as well, but that's something I could pull out into the HTML later if I needed to.

Styling:

The exciting thing about the rotating gallery is using CSS classes and ids with the CSS3 transitions to create neat effects just by swapping the classes and ids of the divs.  It makes putting neat transitions in your website a lot of fun and fairly easy to do.  I know this isn't new to a lot of web developers, but if you're somewhat new to web design and development, this stuff can be pretty exciting.

Let's take a look at the CSS file and see how it's doing a lot of the work for me:

You can see here, we wrap our boxes in a holder with position relative, so that we can use absolute positioning.  Also the neat work is done with the transition tags.  All it's saying is that between changes in the class to modify position, we're going to give all transitions a .5 second ease between the divs, and the browser will do all the animation work for us.  Neat!

Here we're specifying the size and dimensions of our Previous node, our Next node and our Hidden nodes.  You can think of this gallery like a linked list.  We're going to be rotating through images, and the first image will always appear after the last one and vice-versa. For this gallery I wanted to make it seem like the pages were fading away so I set the opacity to 50% and the size to 50% for the images to the side.  

You'll also see I set the Z indices up so that the main one was the highest and the next/prev ones were in between and the hidden nodes were the lowest.

Note:  It's important when you set your Z-Indices, to keep them positive if you want the divs to be clicked on.  If you don't it's likely click functions won't work because they'll be behind the main layer of the page.

So now all we have to do is link the divs together and add some click functions!

Script:


The first part of the script is the set up.  I call this to push each ID into an array for my divs and set them all up to be ready to be called by the rotate functions.  I set all the divs to invisible and hidden and add the on click functions to each of the divs (only called when visible and not the current one), so essentially only the previous and next ones can be clicked (but I could expand this out later if I wanted).  Then I call the rotate function to set up the screen properly.

The next is our rotate function.  This calls out to a few other functions that I've made, the GetPrevious and GetNext functions just check the next and previous number based on the total number and the current number (so GetPrevious(1, 5) is 5 so that we have a linked list) and then the RotateNext and RotatePrevious functions change the classes on the divs accordingly.

I'll show you a snippit of the RotateNext function so you can get an idea of what it's doing.  Basically I need to shift all relevant divs, which means, 2 before, 1 before the current and 1 after are the relevant divs, since all others will remain hidden.

Then I set the currentShow (the currentDiv, not sure why I named it that) and we're good for this rotation!  Then if our current one isn't the one we wanted to click to, it will continue to rotate until we reach our destination.

So that's a little bit about the Rotating Gallery that I made,  next steps are working on all the pages for the actual games and getting A Slight Anomaly up on it's own page as well as a playable demo, and then testing and pushing to live!  Exciting stuff.

Hopefully this weekend I can power through the game sites so that I can start prepping to push the new website live.

Subscribe to stay tuned into the updates and to learn some neat tips and tricks about web development!

Cheers,
Jason C.

Website Press Page Finished - Custom Gallery instructions

I just got done with the new Press Page for the new website.  There are some neat things I'm doing here so I figured I'd share some of it for newer web designers and developers who might read the blog.

The first is that every page is dynamically loaded, so when you click a button link, the page is loaded from a separate HTML page and then included in the main div that stores the page information.  This means there aren't many redirects which saves resources, but also means you can't link to pages directly, you have to navigate to them.  I'll have to look into adding a way to load a page with additional header information.

The second cool thing, (a little thing, but I like it a lot) is creating a fixed position menu bar for the Press page that sits at the top of the page, but won't overwrite the header image.  I'll show the code for this as well.

And the final thing is a custom gallery for the Press page that loads images and titles from an XML page and displays the images in a miniature gallery bar and then when they're clicked displays them and the title in the main frame.

Dynamic Loading

So for each page, I'm dynamically loading the HTML into the main "div" that holds all of the page information.  This div sits below the header and is emptied out and replaced with current page information on button click.

Here's a snippet of what the button is doing when clicked to load the Press page:

I use JQuery to load the page in and then when it finishes, I add in the functionality for the Menu Item slide panel, and the Gallery.

Sliding Menu Panel

This brings us to the next neato thing, the Sliding Menu Panel.  As you can see below in the next two screen shots the menu panel sits below the header, but as you scroll down, it locks to the top of the screen.

Sits Below Header

But Locks to the Screen as you scroll down

This isn't tremendously complicated, but it involves adding a little bit of JavaScript in order to get it working correctly.

As you can see in this snippet, I'm changing the CSS dynamically depending on the scroll location of the page.  And if my current page is not the Press page, I remove the scroll listener so it doesn't affect anything I don't want it to.

Custom Image Gallery

The final cool thing I did for this page is create a custom image gallery.  I wanted to be able to add images quickly and without touching the JavaScript or HTML and I wanted to include information with each image (such as a title), so I'm storing the image information in an XML and retrieving it when the Press page loads.

In the second screen shot above, you can see the gallery as it's loaded.  Each miniature image updates the main image frame when clicked.  (Also you'll notice I customized the scroll bar's CSS, there's a nice article on that here).  

First let's take a look at the XML structure.  It's not very complicated and you can add whatever kinds of data you want and you can use JSON if you prefer, but I think for simple things like this, XML is much easier to read.

Once I had my structure set and my XML page filled with all the titles and image locations I needed, I then wrote a simple script to load in that data (in the function loadGallery() that you saw earlier).

Here is the script, including the function I used to update the main gallery frame:

I won't go into too much detail, the script uses an AJAX get request to grab the XML and then runs through each image's data to create a list with each miniature gallery item, including an on click function to update the main gallery frame with the image when clicked.  Then I get the first image from the XML and load that into the larger frame.  Then it's just CSS to get it looking right.

These are just some little tips and tricks I thought you might find neat.  The site is coming along great, all that's left is a bit more security checks (like encrypting passwords in my database, HA!) and the Games page.

Of course the Games page will have links to pages with information about each game, so that's still quite a bit of work, but it's all relatively simple stuff.  The Press page was the big one for styling and formatting work, but it's looking alright.

Also, the website will release with a brand new game to try out!  It's a VERY rough demo, but I want to get some feedback on the game before moving further with it.

That's it for now!  I'll keep the blog updated as I finish up the website and get it up and running.

Cheers,

-Jason