Wednesday, September 23, 2015

Making a Real Time Multiplayer Online Game in NodeJS (Part 2)

In Part 1 of this blog series I walked through the idea and overall Architecture of QuadPong, a four player online pong game made in NodeJS and HTML5.  Now we're going to take a look at the server side code, walk through each portion of the code and review key portions of it.  We'll start with the entry point for the application, server.js and then jump all the way down to the deepest parts of the code, looking at the game object and physics classes.  We'll slowly work our way back up through the game loop to the socket layer, where we'll close out by reviewing how the server talks to the client.

Note:  This is quite a long post, so I do encourage you to get fresh cup of coffee or tea, get comfortable, and skip around to the interesting parts if you need to.  Also, you can view all of the source code on GitHub, which should be of help when going through this post.

Server.JS -- /server.js

If you've done any coding with NodeJS or Express, server.js is pretty self explanatory.  Only 30 lines long, server.js simply sets up our http server, connects to and routes our socket connections to our SocketHandler script.  Express also has a static function making it very easy to server our public folder with just a line of code.

The hosted site ( is served behind NGinx as well, but I chose to use Express to host the static files as it ends up being easier to configure for a small project like this one.

That's it for the entry point to the application.  We'll now go all the way down to our Math, Physics and Game Object libraries and take a look at what makes up the core of the application.

Libraries -- /app/lib


First let's look at physics.js.  Primarily a Vector2 library (Math/Physics, whatever!) it also contains the Bounding Box class used by our Game Objects.  

Disclaimer:  I am not a mathematician and never took Linear Algebra in college, so my knowledge of Vector Mathematics is limited to what I've taught myself on Khan Academy and from what I've needed to know from working with 3D and 2D programming in the projects I've done.  I'm sure there are much more efficient formulas to do some of the things in this library.  However, it does do the things I need it to.

So, with that out of the way, I went with a system that used Vector2 as a coordinate, rather than describing the beginning and end of the vector.  All Vector2s are assumed to start at (0,0).  I kinda just made it up as I went, so its not a full fledged Vector2 Library and if you want to make it one, you'll need to do some work to it.  This is one area that could use a lot of refactoring and could be made much better.

So down all the way in /app/lib/physics.js we see on line 61, the Vector2 Class constructor.  Vector2s are basically x and y coordinates, with tons of functions to return the data needed.  

In the preceding lines you'll notice a bunch of private functions, like getDirectionFromVector and getVectorFromDirection, and you can take a look at those, but they're mainly just assisting functions for the main Vector2 functions.

There is one function before the constructor that I'd like to take note of: degreeAdjuster.  It's been a long time since I've done trigonometry, and I'm sure there's a better mathematical solution for this, but a simple way to adjust for the way Math functions round when given a value is to determine which quadrant you're in, and adjust the output degree accordingly so that you get an accurate direction from 0 to 360 degrees.  In other words, this makes sure that when I get a direction, it's always a degree between 0 and 360 which is what I need for the other functions.  It is a little hacky, but get's the job done.

In the lines following the Vector2 constructor, you'll find the functions tacked on to the prototype so that they can be called on a Vector2 object.  There are a lot of functions and they are primarily split into two types: those that mutate the object they are being called on, and those that return a new Vector2 object.  Because in JavaScript passing in an object is always by reference, I kept the idea that in general the functions should mutate the Vector2 object being used to call them.  However there are some functions that do not follow that convention, and the whole library could honestly use a significant refactor.  I also included a copy function which simply clones the object it's being called on. You'll also notice, most of the functions will return this so that they can easily be chained together.

On the left, are the normalize, copy, and getMagnitude functions.  Normalize is used to take a vector of any size and mutate it to its unit vector, having a magnitude of 1.  This function also uses a helper function roundToPlace, which simply takes a number and rounds it to a specific number of decimal places (in this case 2).  

Copy is used quite often, and is a very useful function for making a clone of a Vector2 to mutate and return for uses where you do not want to affect the original.  You'll see this used a lot in the classes file.

There is also the getMagnitude which is just a wrapper for our private getMagnitudeFromVector function, which given a vector, will calculate and return its magnitude.

Finally let's look at a few more conspicuous functions, rotateDegrees and reflect, which may not be quite as intuitive as some of the other functions.  rotateDegrees is used to rotate the current Vector2 around the origin a certain amount and then returns a new Vector2 based on that amount.  This is one of those functions that doesn't quite fit the pattern of some of the others and could use a rework.

The library also has a reflect function which given a normal vector, will reflect the current vector across that vector.  Think about a ball bouncing against a wall at an angle.  If the normal is the vector coming out from the wall, then it uses that vector to calculate the new vector of the bouncing ball after it hits the wall.  This function is primarily used when calculating the bounce of the ball against other game objects.

So enough about the Vector2 class, you can look on GitHub if you want to review the other functions, but they're fairly straightforward.  

Next in our Physics.JS file we find the BoundingBox class.  This is a simple rectangular bounding box used to determine our collisions in the game.  You can examine the constructor on line 251, with a generateBoundingBox function below it, which generates the x1, y1, x2 and y2 values from the position, width, and height.

I won't spend a ton of time talking about the bounding box as it is fairly simple.  There is an update function and the intersects function, which is used to determine if two bounding boxes intersect, seen here.

Other than that there are a few degree to radian conversion functions, helper functions and then it's packaged it all up in module.exports.

There are also couple other scripts of note here in our library: a math_extentions script which when required, will add a Clamp function to the Math class which simply clamps a value between a min and max, and a helper.js script which simply has a few guid creation functions.  Alright!  Now onto our classes.js script!


Classes.js contains three classes:  GameObject, which is the main parent class for our hierarchy of classes, and the Ball and Paddle classes which are children from it.  The GameObject class contains a Vector2 position, a Vector2 velocity, a name and a BoundingBox.  It also has one function at the moment: intersects, which checks another GameObject's bounding box against its own using the function from our physics.js script.

Below is also the Ball class.  It inherits from GameObject (on lines 39 and 45) and has a few of it's own fields like colliding (boolean check if it's currently colliding with something), colliding_with (GameObject reference of the other object in a collision), and game (a reference to the game the object is in).

Above is also the updatePosition function, which given a delta time, updates the position based off of the velocity, updates the bounding_box and checks if it is still colliding with anything after it's moved a bit.  There are a few more functions: 
  • bounce_x and bounce_y - swaps the x portion of the velocity and the y portion respectively
  • reset - sets the ball position to (0,0)
  • start_velocity - sets the initial velocity at the start of a point
  • checkBounds - determines if the ball is out of the playing area and scores or bounces accordingly.
Note: One "to do" I have is when refactoring in the future, to be consistent with my snake (underscore) and camel cases.  I mix and match them for my function names and it's not the best looking.

There's also one more function we'll look at for the Ball class, bounce_against, which I feel deserves a look at.

Bounce against takes a game object that it is colliding with and checks if it is already colliding with something.  If not, it will collide with the game object and bounce against it, reflecting over the normal vector.  In addition, I wanted to add a portion of the paddles velocity to the ball, as well as the vector from the center of the paddle to the ball, to allow the player to have some control over the direction of the ball.

At the end of the function I added other_velocity, which is the velocity of the paddle the ball is colliding with, and the additional_velocity, which is a calculated velocity vector based on the position on the paddle where the ball was hit, and then I normalized the vector and set its magnitude to the original magnitude so the ball speed is not changed throughout the game.

Finally in our classes script is the Paddle class on line 161.  The Paddle class is the player in a game, it holds the player's remaining life in a game, the action a player or computer is taking, as well as a bunch of other variables that we'll look at in some of the functions.

The Action is the primary directive that a player on the client needs to send to the server.  Each Paddle has 3 action states:  Move Left ("L"), Move Right ("R"), and Nothing ("N").  These are updated from the client or the computer action function, seen below.

Players can be either human or computer, and if the player is a computer, it will be directed in the game loop to calculate its current action based on the ball position.  It's not complex AI by any means, but for the purposes of this game, it works quite well.

The other functions are there to simply calculate and update the position of the paddle based on the action, and update the width of the paddle to shrink it when a point is scored against that player.

In addition there are a few private helper functions used to calculate a starting ball direction, the positive_vector for a player which is used to determine which direction is "Right" in respect to the normal vector being forward, and a function to help generate the appropriate bounding box for a given player number.

That's it for the classes script.  This is a simple game, so there are only have 2 main classes: the ball and the player.  For a larger game, I'd recommend breaking these down into smaller files as this could easily get unweildly in a single file.

The Game Loop -- /app/game_loop.js

Now we're going to hop up a level to the Game Loop.  The game loop contains the bulk of the game flow and logic on the server.  Game extends the Event Emitter object, so that it can use those functions to emit updates to the client once per loop.

In /app/game_loop.js we can see the constructor and main logic for our game loop class.  On line 10, is the initial object for the game that houses all of the data and functions.  It sets up a few variables such as the game id, and some references to the players and ball object which are in the game.

The main game loop starts on line 28, in function gameLoop.  This function is run as the update function.  This game is built in NodeJS, so it's prudent to be careful of how much processing is chucked on the event loop, and while the game is doing all of the calculations synchronously, I didn't want to repeatedly call the gameLoop over and over without any break for the event loop to work on other things, because that would eat away all of the CPU I have available for the application and it wouldn't be able to run multiple games per process.

So instead of calling the loop again instantly I used setTimeout to queue another call of the function on the event loop stack a specific time later (generally 15 ms from the time the current loop was called which equates to roughly 60 fps, this time is set in the config).  The time between when this loop runs and the next call becomes our delta time as set on line 29.

The entirety of the game loop is pretty self describing within the function:
  • Set our Delta Time and start time
  • Move our Players
  • Move the Ball
  • Check for collisions between the players and the ball
  • Check if the ball is out of bounds
  • Set our end time
  • Check if the game was won
That's the main game logic.  I also added a check to make sure that any loop over 30 ms gets logged so I can see if the server is getting slow as well, and then the function emits out the player and ball data to the socket handler, and checks if the game is over by checking the scores.

Finally it queues up another loop if the game is still active (status === 2) a few milliseconds later.

There is a wide range of both private and public functions on the game object, but we'll just take a look at a few of them, which are of note.

Move player paddles, move ball and check collisions are pretty self explanatory; they iterate over the stored objects, updating their positions, and then checking bounding boxes of the player against the bounding box of the ball.  

The end of game function I'll show since it emits the survivor out to the server side socket handler, which we'll look at in a bit.  This function checks the life of each player and if only one survivor remains, that player is emitted out to the client in a notification that the game is over.  The game status is then set to 3 indicating that the game is done, and can be cleaned up.

For public functions, some notable ones include, addPlayer and addComputer which, as you can imagine, add a player and a computer respectively to the game, and gameStart which puts the game into play and begins to run the game loop.  As you'll notice in the gameStart function, it sets the status to 2, which is used by the game loop to determine whether or not to continue to run.

Another function of note is the updatePlayerAction function, which when the game is started, will change the action of the player in the game.  This is the way the client sends data to the running game and how state among the players is changed.

And then the on point event will score against players and notify the client.

Thats pretty much all there is to the game loop.  It acts out a very basic game system, updating each actor in the scene, and calculating all of the simple physics that happen.  For a small game like this, the logic is fairly compact, so it don't use much processing power on each loop.

If you wanted to take this and do some complex pathfinding algorithms, it would be worth offloading that to a separate thread with a C or C++ library for NodeJS.  I haven't had the need to create my own library yet, but I know it's possible, since it's what makes the callback model work.

The last main section of code is the socket handler for the server: server_sockets.js.

Socket Handler -- /app/server_sockets.js

The socket handler script, server_sockets,js, is the main communication layer for the server side.  It primarily talks with the parallel script on the client side, socket_handler.js to send web socket communications back and forth via emitted events.

When a player connects, they hit handleSocket, which is triggered by when the client connects via our server.js connection event.  The client will then emit a "new_player" event, which is caught in the below function on our socket.

In this event, the handler checks if any games with the game_id passed in have a player with the player_id passed in, and if so, uses a bit of reconnection logic to determine which state the client game should be in, and what players are in that game.

The socket will then emit "reconnect" with the data, letting the client know it's reconnecting and it should take appropriate actions.  The player is added to the socket "room" for the game they are rejoining, so they can receive the appropriate events.  It'll also emit a "sync_players" event, to make sure the client is aware of all players in this game.  However if the player is not reconnecting, it'll simply create some credentials for the player, namely a player ID #, and send that back to the player and treat them as a new user.

There is also have a disconnect event, a leave game event, and a start game event which all are fairly self explanatory.  When emitted a game id for leave and start, the handler will either drop the player from the current game or try to start the waiting game that the player is in, and if it has enough players, start it up.

The next event below those three is our join game event.  In the current iteration of the game, a player cannot specify which game they would like to join.  They simply request to join an open game, and the server will find an empty game in the games list to pair them to.

Here the socket server searches for a game_id that has open slots, adds the player, and then emits back out a bunch of data.  The logic is as follows:
  • First, have the player's socket join the game room broadcast
  • Emit the game settings to the player
  • Send back information about the game they are joining
  • Emit to all players via io.emit (a broadcast) the new game list data
  • Emit to the entire room for that game the new player joining
  • If the game is full, emit back to the player trying to join that the game is full
And if there is a server error the handler notifies the player of the reason for the error.

There are a few more events which handle: updating actions, creating games, adding computer and human players, requesting games lists and so on.  The rest of these events are pretty straight forward.  It either modifies the game or updates the client back with information.

Other Scripts of Note

So that's the bulk of the back end portion of the application.  In addition to the main scripts, there is also have a configuration folder in /app/config which holds our game settings configurations, as well as some mocha tests in /app/test.  Currently there is just a test to make sure the physics library works as expected, but more should be added to ensure our scripts work properly.

I hope you enjoyed this lengthy post about the server side component to the game.  The next post should be much shorter and will talk about the client side, the canvas element, the socket handling, and some other small libraries I used to handle input and the like.

Feel free to follow me on Twitter, and check out the GitHub with all of the code for you to fork and build on!

Monday, September 21, 2015

Making a Real Time Multiplayer Online Game in NodeJS (Part 1)

So, I have this bad habit of starting too many side projects.  This is one of those projects.  Having never built an online multiplayer game, I wanted to make a very simple, real time game using what I use every day at work:  NodeJS.  And so I did.  I built a game called:  QuadPong.  The source code is available on GitHub and is free to use under the ISC Open Source License.

This series of posts is a guide to the open source code, showing what I did, how and why I did it, and how you can take it and make something better.  We're going to take a look at the architecture, the back end and the front end and step through exactly how it all works together.  So first, let's talk about the idea, the overall architecture, and what I was trying to accomplish.

The Idea

I wanted to learn more about network coding and synchronization, and create a real time game where players could play against each other with the game run on the server, and displayed in the browser.  For this project I chose to create my own engine, libraries, and structure; excepting a few small libraries I used on the front end.

QuadPong is a four player Pong Game, where each player takes control of a paddle on one side of a four sided arena and must prevent the ball from hitting that wall.  Simple, yet slightly more complex than a standard pong game.

As the players lose points, their paddle gets smaller until they are removed from the game and the ball will simply bounce against their wall. Thus, players are removed one by one until one remains: the winner.

For the network side of things, the player simply passes back state to the server, updating the server of what actions it's doing currently, and receives updates from the server with all of the Game Object positions and bounding boxes to render in the HTML5 Canvas.  It made synchronization fairly easy, as I didn't have a multitude of objects to render, so I could simply pass back the entire state of the game.

The Architecture

I wanted to keep the game simple, with high IO and transferable libraries across the client and server, so I went with a full JavaScript stack, something I'm familiar with at work.

QuadPong is built in NodeJS, using Express and Socket.IO to manage the web socket connections.  In addition I use a little bit of JQuery on the Client, as well as a library called KeyPress.js, which handles input and keystrokes and a snippet of code from Stack Overflow to create a HiDPI Canvas.  The hosted game also sits behind NGinx as a reverse proxy server, which points to the application.

The general application architecture is structured something like the image below.  First in the center we have our web and socket server.  The Socket server handles all requests for the static files (index.html) as well as maintaining the web socket connections of the client.  It spins up new socket rooms and games on creation, as well as handling all communication to and from various clients.

Next we have the client side, where we have our canvas where the game is rendered as well as our client side socket connections.  The Client handles all of the key strokes and actions of a player, passing in state data of the player ID to the server.

The Client side notifies the Socket Server of all events, information, key presses, game creation and join events, and the Socket Server will emit back the corresponding response tag.  For instance, when the client emits a "join_game" event.  The Socket Server will correctly find a game for the player, add then to that game, and emit a matching "joined_game" event back at the Client as well as all other players listening in on that game room.

When a player creates a new game The Socket Server spins up a brand new Game Loop instance and stores a reference to that game in memory.  It also creates a new Socket.IO Room for the player who created the game, as well as any joining players, to join and listen in on relevant events for the game.

The Game Loop instances are the final part.  Each one is a separate game, playing out on the server with which the Clients simply notify the game of their states, and receive updates from the Game Loops.  Game Loops make use of a variety of custom built libraries, including a Class Library of game objects for the game, as well as a Physics Library, with a Vector2 Class I built for the game, and a BoundingBox class.  We'll look into these more in Part 2, when I go into more depth about the Server Side code.

The Game Loops are designed to run off of a minimum delta time between each loop.  This lets us tweak the speeds of the games, and increase the minimum time between loops if the server starts to get overloaded.  Once an event loop has completed on the server, the Game Loop will queue up another function call on the main NodeJS Event Loop some time later (typically 15 ms).  This allows us to create the optimal frame rate we desire, while allowing the server to run as many games as possible.

My Goal

The goal of this project, and series of blog posts is to provide a base project to build real time online games for the web.  These posts are targeted toward intermediate web and game developers looking to start making real time games.  Mostly, I wanted to practice socket connections, and see exactly how much work an online multiplayer game would take at a very base level, including building the engine.  The answer is that it really didn't take much time, but the game is still a ways from being a full fledged game.

I still need to add security features, fallback features, the ability to scale the project, memory clean up, etc.  But at a base level, it works.  You can log in and play with other people if they connect to your game, and play through to a winner.

So I've got the basic build of the code up, and you can see it, download the code, fork the repository and build on it.  Make your own real time games!  And I'll be polishing up the code as well since it's a portfolio piece for myself.

The next post will be about the back end logic: how did I architect it, what does it do, and how it could be made better.  We'll go through the code step by step, so you can understand what I was trying to do and how to replicate and modify it.  After that we'll take a look at the front end, how to render our server side data, handling key inputs and more.

So stay tuned, and feel free to follow me on Twitter, as well as check out my GitHub account and pull the code down!

Tuesday, September 1, 2015

League of Legends API Challenge Entry - PowerSpike

On Monday, August 31st, I could enter no more code.  It was done, complete, or as much as it could be with the time invested.  I had finished my entry for the Riot Games, API Challenge #2.  I had created and submitted PowerSpike.

The API Challenge I took part in was to create a web application that referenced data sets from before and after a massive itemization change that Riot made to their game, League of Legends.  When I read the requirements, it sparked an idea.  What DID the data look like for champions as they got kills throughout the game?  How many kills in each minute of a game, does a particular champion get on average, and how did those item changes affect that?

And so I set out to satisfy that curiosity.  I began work on an application called PowerSpike, using NodeJS to serve up a web server, as well as run the parser which would collect the data.

Those two parts contain much of the application code.  

The Riot API Parser

First, the parser runs through each match, pulling down the data from the API and reading in the kill data for various champions in each match.  I chose to use the four data sets for the North American data, since those are the servers I play on and I wanted to limit the amount of data to specific regions.

When the parser starts up, it checks each of the matches in the riot test data files against a list of match IDs that have been parsed already and if they haven't been parsed already, it queues them up to be run.  Then it gets the data, grabs the kill times for each champion and stores that data in MongoDB as kills per each minute.

It does this slowly, waiting between each request so we don't hit rate limits by the API.

The Web Server & Client

The other half of the application is the web server which exposes a few endpoints for data retrieval from MongoDB, as well as the client static pages, which display the data in line charts over time throughout a game.

Chart.JS is the library for displaying data that I used and it's pretty good at the basics.  Each of the four data sets gets compared to one another.  I also modified the chart to allow the user to toggle datasets with the legend as well, since the basic UI can be cluttered.

There are some very interesting data points for a few champions as well.  For example, Ahri gained a significant boost in early game kills for normals after the patch.  This could be due to the much earlier Needlessly Large Rod that players could now purchase at an earlier time.

But either way, the data points are pretty interesting. and there are a few champions like Kayle and Teemo that have some pretty interesting data to look at.


I decided to host the project on a web server, so I spun up a new Digital Ocean droplet and purchased at for 1 dollar for the first year.  I highly recommend Digital Ocean as I run many of my projects off their boxes.  You get full control of the VPS and it's great for being able to do all sorts of interesting projects.

It was a pretty fun challenge, and it's now in review for the next month, so we'll see how it goes!

You can view the application at:

And the full source code is at my GitHub:

Feel free to check it out and let me know what you think!

Jason C.

Monday, June 22, 2015

Article Curator - Web Scraping with Casper JS and Node.

Article Curator

I was on vacation recently and had some time to start up a side project, and so I created a custom article curator for a website I'm working called The Tamriel Underground.  I wanted to automatically scrape sites and pull in articles that I could link to and show on my site, so that I could have automatic content populate the articles section.

And it works fairly well for the initial build.  I used CasperJS for the web scraper and NodeJS to spin up a web service which would save the data to a Postgres database.

You can view the whole project on my Github.  It's just 2 JavaScript files, one for the Scraper and one for the Web Server.  It's a fairly small project at the moment, but I plan to build a few more scrapers for other sites to pull in data.

The Code

Both files are only about 200 lines of code combined, so it's not a ton of code to walk through, but this was my first time using Casper and Phantom so it took me a bit of work for those 200 lines.

The web server gets run constantly and stays up using node forever, and the web scraper is run as a cron job every 2 hours.

First let's take a look at:


The goal of this script is to access the Elder Scrolls Online Website, log in through the Age Gate, and then scrape the Patch Notes site for the links to the patch notes.  Then we iterate through those links to pull down the html used in the article.

So at the top of our file we have our declarations:

Here, we specify our urls that we're going to be accessing, we initialize Casper and we create an array and object to store our data as well as set a few settings.  Pretty straight forward nothing special here.

Next, we're going to skip over a few lines down to line 87.

Here is the starting point for Casper.  We open up our age_gate_url which points us to the page we need to fill out before going any further, then we wait for the page to load by waiting for the selector to show up, and then we fill out the form.

Next we open up the page where we want to pull our patch note links from and call out to our getPatchNoteLinks function which we'll look at later, and evaluate that within the page.

Finally we run Casper.  This is what actually makes all the steps we've declared so far happen.  And once all those steps are done, and we have all of our patch notes links, we make the call to getNotes, which is the callback after Casper is finished running.

Now we start parsing those links we've accumulated.

getNotes iterates through each link, calling at the end of the link processing, recursively calling itself at the end of each run, until all links are processed, in which we call out to our save function.

Above you can see the three functions we've used so far.  The first one, which we ran in the evaluate statement is the middle one, getPatchNoteLinks.  Now this function is run inside the loaded page.  It's just like client side javascript or if you ran it in your console.  And it will return the value back to our PhantomJS/CasperJS 'scope'.

You see the evaluate statement used again below in the curateLink function as well.  Here we're taking a link, we're using evaluate to grab the title from the page and return it, then we grab the entire article which is between the article tag, and then we add all of that information to our patch_note_info object.

And finally, once all this is done, we call out to our final function, saveInfo.

The reason that we have two parts to this application is that CasperJS and PhantomJS are NOT node applications, and thus there are no server side database drivers (that I know of) built for them for Postgres.  However we CAN post that data to a separate NodeJS server that will store it for us.

saveInfo does exactly that.  We post our stringified object with all of our data to our NodeJS server and let it do the saving for us.

This also allows us to keep our curators light weight.  We can have a single location that handles saving to the database, and the curators can simply handle the scraping and POST it when done.

So let's look now, at our simple web server.


Again, we have our basic declarations and requires at the top:

For this app I used express, simply because I'm familiar with it and it's easy to get a simple REST server up and running, however there are much lighter weight options out there for something small like this.

We require:

  • express - a nice REST framework
  • pg - this is our Postgres connection module allowing us access to our database
  • body-parser - this allows us to easily parse JSON data posted to our server
  • async - a nice framework for iterating through database calls with different datasets.
And then we set our connection string and start our app.

Then we have the bulk of our web server.

We include our body parser json so we can parse the data sent easily.

Then we have our one endpoints POST /save.  Here we grab our articles, we pull out our sites and push the data to an array.  Then we connect to our database, and check if those sites already exist in the database.  If the sites exist, we'll skip them, otherwise we'll add the data that we don't have yet to the database.

And that is exactly what these two functions do.  insertSites will takes the sites we don't have already stored, and iterate through each one with the async library.  This is where the async library is convenient.  It allows us to easily iterate through asynchronous methods which we need for the database insertion.

And that's all there is to the Curator!

That data then gets consumed by the Tamriel Underground, which is a Django application and converted to a model.  I lined up the model schema in python and the way I'm entering the data and it works quite well.

The articles get added to the page, and you can view them with the plus button.

Right now I'm using the html I get from Elder Scrolls Online as is, but I'm going to create a cleaning function to strip out anything that might be invalid.  Since I'm going to be scraping the html and displaying it on my site, I want to make sure that no one can insert any nasty script tags or links that might cause a security concern.

Since I know the source that I'm scraping I'm not terribly concerned at the moment, but as I add more sources to the curator, I'll definitely want a robust input cleansing module.

So that's pretty much it, feel free to pull the Github repo down and play around with it and let me know what you think.

Jason C.

Friday, May 29, 2015

Green Lit

Well I can't believe it happened, but SBX: Invasion finally got Greenlit and will be on Steam on June 1st!

It's been a long time coming and it's been an awesome experience!  At first when I got the e-mail that SBX Invasion had been Greenlit, I thought it was a phishing scam, so I skeptically opened a separate tab and went to the steam page.  And then it hit me: a game I had made was going to be on Steam, a game that had sat in the Green Light process for almost 15 months.

I got excited.  And then I got to work.  I had to polish a few bugs I knew of, create new builds, get it compatible with the Steam upload system, create new art assets for the store, create a new promo video, and learn how Steamworks Builder worked.

I spent a couple weekends preparing and getting everything together and set the launch date to June 1st.  And now it's almost here.

I don't know what this will mean for SBX: Invasion, but a small part of me feels like I've succeeded, just a tiny bit.  That feels good.  

So if you haven't yet, check out the Steam Page, or check out my Youtube Promo Video

Or drop me a line on Twitter.

Cheers and hope to have some awesome things in store to share soon!

Jason C / WakeskaterX

Tuesday, April 28, 2015

Tamriel Underground

I've been working on a fan site lately; a home for the Thieves and Scoundrels of Elder Scrolls Online to gather and share news.

I present to you, the Tamriel Underground:

Tamriel Underground is a members only site for the Thieves of Tamriel to share tips, tricks, maps and guides.  It will be a place to post Thieving challenges and share your exploits.

Currently only the Maps Tab is active, and I've taken maps from the Rogue's Folio (by reddit users /u/anemonean -- props) and uploaded them and organized them to be refined and searched on.

You can view the maps in a list, and refine them at the top or use the search bar to refine the list down to what you are looking for.  Once you click on the map image you want, it pops up in a lightbox, letting you see the full map.

The website is in it's early phases, and there isn't a lot of content at the moment, but the plan is to create a massive collection of all of the best Thieving tips and tricks and to keep away all those pesky Guardians that will eventually be trying to steal our secrets.

Come join The Underground, where the night is your friend and stealth is your weapon.

Thursday, March 5, 2015

NodeJS - A Little Experiment in Load Testing and Clustering

Load Testing NodeJS on Multiple Cores

Check Out the Git Hub project HERE

Something I wanted to investigate was using Node JS on a multi core system to do CPU intensive applications under high load, so I created a project to do just that.

There are a few ways to use multiple cores in NodeJS, two of which are Cluster (part of the NodeJS API) and WebWorker Threads.

I will preface this by saying I am no NodeJS expert, this project was simply for learning on my part and there may be a better way to go about this with an NGinx set up, but I wanted to share my findings and maybe you'll find it cool too.

The Planning Stages

I started the project out with a simple NodeJS and Express server, which when called with a number in the query, would calculate and return the Fibonacci sequence for that number to simulate high CPU calculations.

My first thought was to use Web Worker Threads to spin up a thread to do work on, and just pass in the data and let it work on a background thread, but this caused issues under high load as too many threads were being spun up (I had no cap at the time) and at the start they were descoping and causing segmentation faults.

I ended up fixing the Seg Fault issue, but even so, they were burning through all of my VMs 2GB of memory and crashing.  

I attempted to create a Worker Pool function to control the amount of threads at any one time, but it quickly got quite complicated.  The idea was to use a queue and simply drop information off and process when ready, but with web requests and waiting on data, this as well got very complicated.

Then I stumbled across the Cluster section of the NodeJS API and found my solution.  With the Cluster API you can spin up multiple instances of your node server on the same port.  So I decided to test if this truly would improve my performance.

Load Testing

I installed loadtest in order to deliver high concurrency testing to my application and began to test the application.  On one branch of my git project I had the vanilla, single threaded NodeJS server which would generate a Fibonacci sequence, and on my other branch I had my project which generated 4 servers on a single port, which would use up the multiple cores.

This was a MUCH simpler solution to building a multi-threaded system within a single event loop and was extremely easy to set up.

So the load testing began.  I called out to my application like so:

loadtest -t 20 -c 32 http://localhost:3030/fib?num=25

I pinged my application using various concurrency values for 20 seconds testing periods.  I tested up from 1 concurrent connection up to 1000 connections.

While I don't believe all four cores were actually being used (as it was on a VM and my computer didn't crash) I did see quite an improvement using the multiple core approach (as one would expect).

The Results

The results from the load test are as follows.  Load test outputs the maximum response times for 50%, 90%, 95%, 99%, and the maximum response time as well as Requests Per Second, Mean response time, and Total Requests.

Computer Specs:  The VM was an Ubuntu 14 VM with 2 GB Ram, 4 processors.

Each request was made with a Fibonacci sequence of 25 for 20 seconds.

NodeJS Single Event Loop

ConcurrencyCompleted Requests50%(ms)90%(ms)95%(ms)99%(ms)Max (ms)Req / SecMean Lat (ms)

NodeJS Multi-Core with Cluster

ConcurrencyCompleted Requests50%(ms)90%(ms)95%(ms)99%(ms)Max (ms)Req / SecMean Lat (ms)

As you can see the results from using the multi-core approach were much better than the single event loop, and using Cluster is super easy to do.  

You can fork the GitHub project here:

The main fork is (well master too, but) hostedVM which has the standard deployment of nodeJS and express, and the multi-core approach is on the hostedVM_multicore branch.

Feel free to test it as well and let me know how your results are with better machines than my very low powered VM.