Monday, September 21, 2015

Making a Real Time Multiplayer Online Game in NodeJS (Part 1)

So, I have this bad habit of starting too many side projects.  This is one of those projects.  Having never built an online multiplayer game, I wanted to make a very simple, real time game using what I use every day at work:  NodeJS.  And so I did.  I built a game called:  QuadPong.  The source code is available on GitHub and is free to use under the ISC Open Source License.

This series of posts is a guide to the open source code, showing what I did, how and why I did it, and how you can take it and make something better.  We're going to take a look at the architecture, the back end and the front end and step through exactly how it all works together.  So first, let's talk about the idea, the overall architecture, and what I was trying to accomplish.

The Idea

I wanted to learn more about network coding and synchronization, and create a real time game where players could play against each other with the game run on the server, and displayed in the browser.  For this project I chose to create my own engine, libraries, and structure; excepting a few small libraries I used on the front end.

QuadPong is a four player Pong Game, where each player takes control of a paddle on one side of a four sided arena and must prevent the ball from hitting that wall.  Simple, yet slightly more complex than a standard pong game.

As the players lose points, their paddle gets smaller until they are removed from the game and the ball will simply bounce against their wall. Thus, players are removed one by one until one remains: the winner.

For the network side of things, the player simply passes back state to the server, updating the server of what actions it's doing currently, and receives updates from the server with all of the Game Object positions and bounding boxes to render in the HTML5 Canvas.  It made synchronization fairly easy, as I didn't have a multitude of objects to render, so I could simply pass back the entire state of the game.

The Architecture

I wanted to keep the game simple, with high IO and transferable libraries across the client and server, so I went with a full JavaScript stack, something I'm familiar with at work.

QuadPong is built in NodeJS, using Express and Socket.IO to manage the web socket connections.  In addition I use a little bit of JQuery on the Client, as well as a library called KeyPress.js, which handles input and keystrokes and a snippet of code from Stack Overflow to create a HiDPI Canvas.  The hosted game also sits behind NGinx as a reverse proxy server, which points to the application.

The general application architecture is structured something like the image below.  First in the center we have our web and socket server.  The Socket server handles all requests for the static files (index.html) as well as maintaining the web socket connections of the client.  It spins up new socket rooms and games on creation, as well as handling all communication to and from various clients.

Next we have the client side, where we have our canvas where the game is rendered as well as our client side socket connections.  The Client handles all of the key strokes and actions of a player, passing in state data of the player ID to the server.

The Client side notifies the Socket Server of all events, information, key presses, game creation and join events, and the Socket Server will emit back the corresponding response tag.  For instance, when the client emits a "join_game" event.  The Socket Server will correctly find a game for the player, add then to that game, and emit a matching "joined_game" event back at the Client as well as all other players listening in on that game room.

When a player creates a new game The Socket Server spins up a brand new Game Loop instance and stores a reference to that game in memory.  It also creates a new Socket.IO Room for the player who created the game, as well as any joining players, to join and listen in on relevant events for the game.

The Game Loop instances are the final part.  Each one is a separate game, playing out on the server with which the Clients simply notify the game of their states, and receive updates from the Game Loops.  Game Loops make use of a variety of custom built libraries, including a Class Library of game objects for the game, as well as a Physics Library, with a Vector2 Class I built for the game, and a BoundingBox class.  We'll look into these more in Part 2, when I go into more depth about the Server Side code.

The Game Loops are designed to run off of a minimum delta time between each loop.  This lets us tweak the speeds of the games, and increase the minimum time between loops if the server starts to get overloaded.  Once an event loop has completed on the server, the Game Loop will queue up another function call on the main NodeJS Event Loop some time later (typically 15 ms).  This allows us to create the optimal frame rate we desire, while allowing the server to run as many games as possible.

My Goal

The goal of this project, and series of blog posts is to provide a base project to build real time online games for the web.  These posts are targeted toward intermediate web and game developers looking to start making real time games.  Mostly, I wanted to practice socket connections, and see exactly how much work an online multiplayer game would take at a very base level, including building the engine.  The answer is that it really didn't take much time, but the game is still a ways from being a full fledged game.

I still need to add security features, fallback features, the ability to scale the project, memory clean up, etc.  But at a base level, it works.  You can log in and play with other people if they connect to your game, and play through to a winner.

So I've got the basic build of the code up, and you can see it, download the code, fork the repository and build on it.  Make your own real time games!  And I'll be polishing up the code as well since it's a portfolio piece for myself.

The next post will be about the back end logic: how did I architect it, what does it do, and how it could be made better.  We'll go through the code step by step, so you can understand what I was trying to do and how to replicate and modify it.  After that we'll take a look at the front end, how to render our server side data, handling key inputs and more.

So stay tuned, and feel free to follow me on Twitter, as well as check out my GitHub account and pull the code down!




Tuesday, September 1, 2015

League of Legends API Challenge Entry - PowerSpike

On Monday, August 31st, I could enter no more code.  It was done, complete, or as much as it could be with the time invested.  I had finished my entry for the Riot Games, API Challenge #2.  I had created and submitted PowerSpike.

The API Challenge I took part in was to create a web application that referenced data sets from before and after a massive itemization change that Riot made to their game, League of Legends.  When I read the requirements, it sparked an idea.  What DID the data look like for champions as they got kills throughout the game?  How many kills in each minute of a game, does a particular champion get on average, and how did those item changes affect that?

And so I set out to satisfy that curiosity.  I began work on an application called PowerSpike, using NodeJS to serve up a web server, as well as run the parser which would collect the data.

Those two parts contain much of the application code.  

The Riot API Parser

First, the parser runs through each match, pulling down the data from the API and reading in the kill data for various champions in each match.  I chose to use the four data sets for the North American data, since those are the servers I play on and I wanted to limit the amount of data to specific regions.

When the parser starts up, it checks each of the matches in the riot test data files against a list of match IDs that have been parsed already and if they haven't been parsed already, it queues them up to be run.  Then it gets the data, grabs the kill times for each champion and stores that data in MongoDB as kills per each minute.

It does this slowly, waiting between each request so we don't hit rate limits by the API.

The Web Server & Client

The other half of the application is the web server which exposes a few endpoints for data retrieval from MongoDB, as well as the client static pages, which display the data in line charts over time throughout a game.

Chart.JS is the library for displaying data that I used and it's pretty good at the basics.  Each of the four data sets gets compared to one another.  I also modified the chart to allow the user to toggle datasets with the legend as well, since the basic UI can be cluttered.

There are some very interesting data points for a few champions as well.  For example, Ahri gained a significant boost in early game kills for normals after the patch.  This could be due to the much earlier Needlessly Large Rod that players could now purchase at an earlier time.

But either way, the data points are pretty interesting. and there are a few champions like Kayle and Teemo that have some pretty interesting data to look at.

Hosting

I decided to host the project on a web server, so I spun up a new Digital Ocean droplet and purchased powerspike.xyz at NameCheap.com for 1 dollar for the first year.  I highly recommend Digital Ocean as I run many of my projects off their boxes.  You get full control of the VPS and it's great for being able to do all sorts of interesting projects.


It was a pretty fun challenge, and it's now in review for the next month, so we'll see how it goes!

You can view the application at:


And the full source code is at my GitHub:


Feel free to check it out and let me know what you think!

Cheers,
Jason C.

Monday, June 22, 2015

Article Curator - Web Scraping with Casper JS and Node.

Article Curator

I was on vacation recently and had some time to start up a side project, and so I created a custom article curator for a website I'm working called The Tamriel Underground.  I wanted to automatically scrape sites and pull in articles that I could link to and show on my site, so that I could have automatic content populate the articles section.

And it works fairly well for the initial build.  I used CasperJS for the web scraper and NodeJS to spin up a web service which would save the data to a Postgres database.

You can view the whole project on my Github.  It's just 2 JavaScript files, one for the Scraper and one for the Web Server.  It's a fairly small project at the moment, but I plan to build a few more scrapers for other sites to pull in data.

The Code

Both files are only about 200 lines of code combined, so it's not a ton of code to walk through, but this was my first time using Casper and Phantom so it took me a bit of work for those 200 lines.

The web server gets run constantly and stays up using node forever, and the web scraper is run as a cron job every 2 hours.

First let's take a look at:

eso_pnote_curator.js

The goal of this script is to access the Elder Scrolls Online Website, log in through the Age Gate, and then scrape the Patch Notes site for the links to the patch notes. The script then iterates through those links to pull down the html used in the article.

At the top of the file are the declarations:


Here, I specified the urls that I wanted to access, initialized Casper and created an array and object to store the data as well as set a few settings.  Pretty straight forward nothing special here.

Next, lets skip a few lines down to line 87.


Here is the starting point for Casper.  The start function opens up the age_gate_url which points to the page needed to be filled out before going any further, then it waits for the page to load by waiting for the selector to show up, and then fills out the form.

Then it opens up the page where the patch note links are and calls out to our getPatchNoteLinks function which we'll look at later, and evaluates that within the page.

Finally the script runs Casper.  This is what actually makes all the steps we've declared so far happen.  And once all those steps are done and the script has all of the patch notes links, it makes the call to getNotes, which is the callback after Casper is finished running.

Now the script can start parsing those links it's accumulated.


getNotes iterates through each link, calling Casper.run at the end of the link processing, recursively calling itself at the end of each run, until all links are processed, in which it calls out to the save function.


Above you can see the three functions we've seen used so far.  The first one, which was run in the evaluate statement is the middle one, getPatchNoteLinks.  Now this function is run inside the loaded page.  It's just like client side javascript or if you ran it in your console.  And it will return the value back to our PhantomJS/CasperJS 'scope'.

You can see the evaluate statement used again below in the curateLink function as well.  Here it is taking a link, using that evaluate to grab the title from the page and returning it, grabs the entire article which is between the article tags, and then adds all of that information to the patch_note_info object.

And finally, once all this is done, it calls out to the final function, saveInfo.

The reason that there are two parts to this application is that CasperJS and PhantomJS are NOT node applications, and thus there are no server side database drivers (that I know of) built for them for Postgres.  However we CAN post that data to a separate NodeJS server that will store it for us.

saveInfo does exactly that.  We post our stringified object with all of our data to a NodeJS web server and let it do the saving for us.

This also allowed me to keep the curator light weight.  I have a single location that handles saving to the database, and then curators can simply handle the scraping and POST it when done.

So let's look now, at the simple web server.

article_server.js

Again, there are basic declarations and requires at the top:

For this app I used express, simply because I'm familiar with it and it's easy to get a simple REST server up and running, however there are much lighter weight options out there for something small like this.

The script requires:

  • express - a nice REST framework
  • pg - this is the Postgres connection module allowing access to the database
  • body-parser - allows easy parsing of JSON data
  • async - a nice framework for iterating through database calls with different datasets.
And then the script sets the connection string and starts the app.



Then comes the bulk of the web server.

First it includes the body parser so that it can parse the data sent easily.

Then it has the one endpoint POST /save.  Here it grabs the articles, pulls out the sites and pushes the data to an array.  Then it connects to the database, and checks if those sites already exist in the database.  If the sites exist, it skips them, otherwise adds the data that it doesn't have yet to the database.


And that is exactly what these two functions do.  insertSites will takes the sites the database don't have already stored, and iterates through each one with the async library.  This is where the async library is convenient.  It allows easy iteration through the asynchronous methods which are needed for the database insertion.

And that's all there is to the Curator!

That data then gets consumed by the Tamriel Underground, which is a Django application and converted to a model.  I lined up the model schema in python with the way I'm entering the data and it works quite well.

The articles get added to the page, and you can view them with the plus button.



Right now I'm using the html I get from Elder Scrolls Online as is, but I'm going to create a cleaning function to strip out anything that might be invalid.  Since I'm going to be scraping the html and displaying it on my site, I want to make sure that no one can insert any nasty script tags or links that might cause a security concern.

Since I know the source that I'm scraping I'm not terribly concerned at the moment, but as I add more sources to the curator, I'll definitely want a robust input cleansing module.

So that's pretty much it, feel free to pull the Github repo down and play around with it and let me know what you think.

Cheers,
Jason C.

Friday, May 29, 2015

Green Lit

Well I can't believe it happened, but SBX: Invasion finally got Greenlit and will be on Steam on June 1st!



It's been a long time coming and it's been an awesome experience!  At first when I got the e-mail that SBX Invasion had been Greenlit, I thought it was a phishing scam, so I skeptically opened a separate tab and went to the steam page.  And then it hit me: a game I had made was going to be on Steam, a game that had sat in the Green Light process for almost 15 months.

I got excited.  And then I got to work.  I had to polish a few bugs I knew of, create new builds, get it compatible with the Steam upload system, create new art assets for the store, create a new promo video, and learn how Steamworks Builder worked.

I spent a couple weekends preparing and getting everything together and set the launch date to June 1st.  And now it's almost here.

I don't know what this will mean for SBX: Invasion, but a small part of me feels like I've succeeded, just a tiny bit.  That feels good.  

So if you haven't yet, check out the Steam Page, or check out my Youtube Promo Video

Or drop me a line on Twitter.

Cheers and hope to have some awesome things in store to share soon!

Jason C / WakeskaterX


Tuesday, April 28, 2015

Tamriel Underground

I've been working on a fan site lately; a home for the Thieves and Scoundrels of Elder Scrolls Online to gather and share news.

I present to you, the Tamriel Underground:


Tamriel Underground is a members only site for the Thieves of Tamriel to share tips, tricks, maps and guides.  It will be a place to post Thieving challenges and share your exploits.

Currently only the Maps Tab is active, and I've taken maps from the Rogue's Folio (by reddit users /u/anemonean -- props) and uploaded them and organized them to be refined and searched on.


You can view the maps in a list, and refine them at the top or use the search bar to refine the list down to what you are looking for.  Once you click on the map image you want, it pops up in a lightbox, letting you see the full map.


The website is in it's early phases, and there isn't a lot of content at the moment, but the plan is to create a massive collection of all of the best Thieving tips and tricks and to keep away all those pesky Guardians that will eventually be trying to steal our secrets.

Come join The Underground, where the night is your friend and stealth is your weapon.




Thursday, March 5, 2015

NodeJS - A Little Experiment in Load Testing and Clustering

Load Testing NodeJS on Multiple Cores

Check Out the Git Hub project HERE

Something I wanted to investigate was using Node JS on a multi core system to do CPU intensive applications under high load, so I created a project to do just that.

There are a few ways to use multiple cores in NodeJS, two of which are Cluster (part of the NodeJS API) and WebWorker Threads.

I will preface this by saying I am no NodeJS expert, this project was simply for learning on my part and there may be a better way to go about this with an NGinx set up, but I wanted to share my findings and maybe you'll find it cool too.

The Planning Stages

I started the project out with a simple NodeJS and Express server, which when called with a number in the query, would calculate and return the Fibonacci sequence for that number to simulate high CPU calculations.

My first thought was to use Web Worker Threads to spin up a thread to do work on, and just pass in the data and let it work on a background thread, but this caused issues under high load as too many threads were being spun up (I had no cap at the time) and at the start they were descoping and causing segmentation faults.

I ended up fixing the Seg Fault issue, but even so, they were burning through all of my VMs 2GB of memory and crashing.  

I attempted to create a Worker Pool function to control the amount of threads at any one time, but it quickly got quite complicated.  The idea was to use a queue and simply drop information off and process when ready, but with web requests and waiting on data, this as well got very complicated.

Then I stumbled across the Cluster section of the NodeJS API and found my solution.  With the Cluster API you can spin up multiple instances of your node server on the same port.  So I decided to test if this truly would improve my performance.

Load Testing

I installed loadtest in order to deliver high concurrency testing to my application and began to test the application.  On one branch of my git project I had the vanilla, single threaded NodeJS server which would generate a Fibonacci sequence, and on my other branch I had my project which generated 4 servers on a single port, which would use up the multiple cores.

This was a MUCH simpler solution to building a multi-threaded system within a single event loop and was extremely easy to set up.

So the load testing began.  I called out to my application like so:

loadtest -t 20 -c 32 http://localhost:3030/fib?num=25

I pinged my application using various concurrency values for 20 seconds testing periods.  I tested up from 1 concurrent connection up to 1000 connections.

While I don't believe all four cores were actually being used (as it was on a VM and my computer didn't crash) I did see quite an improvement using the multiple core approach (as one would expect).

The Results

The results from the load test are as follows.  Load test outputs the maximum response times for 50%, 90%, 95%, 99%, and the maximum response time as well as Requests Per Second, Mean response time, and Total Requests.

Computer Specs:  The VM was an Ubuntu 14 VM with 2 GB Ram, 4 processors.

Each request was made with a Fibonacci sequence of 25 for 20 seconds.

NodeJS Single Event Loop

ConcurrencyCompleted Requests50%(ms)90%(ms)95%(ms)99%(ms)Max (ms)Req / SecMean Lat (ms)
648891135169180245286429150
20081103361276135833007364405480
1000817812353726762615726162564092510

NodeJS Multi-Core with Cluster

ConcurrencyCompleted Requests50%(ms)90%(ms)95%(ms)99%(ms)Max (ms)Req / SecMean Lat (ms)
64150097015519332756075080
200144982524765677561062596330
100014923117219542538361876417181350

As you can see the results from using the multi-core approach were much better than the single event loop, and using Cluster is super easy to do.  

You can fork the GitHub project here:  https://github.com/WakeskaterX/NodeThreading

The main fork is (well master too, but) hostedVM which has the standard deployment of nodeJS and express, and the multi-core approach is on the hostedVM_multicore branch.

Feel free to test it as well and let me know how your results are with better machines than my very low powered VM.

Cheers,
WakeskaterX

Tuesday, January 27, 2015

PlayCrafting Boston Winter Expo - February 24th!

Hey All!  Exciting things have been going on the past few months, such as learning new web Technologies and taking part in GGJ 2015 and all sorts of exciting things!   But, this post isn't about that, it's about...

The PlayCrafting Boston Winter Expo!



Stop by at 6pm at the Microsoft NERD center in Cambridge on February 24th, 2015 and check out tons of awesome local indie developers and their games.

You can sign up at the event bright page here.

I'll be there showing off SBX: Invasion and giving away free game related goodies!

Sign up before Feb 23rd to get a discount and come hang out with your local indies!

-Jason C.