Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts

Wednesday, May 10, 2017

BitBit - A little compression library

Ever needed to compress a bunch of settings in a JavaScript object down to fit in a couple of bytes worth of data?  No?  Well that's okay I didn't either until recently.  While working with some embedded systems I needed a way to take simple number and boolean settings in an object, where I could store them in a readable format, and pack those down into under 2 bytes to be sent to and stored on a memory limited device.

Now, bit masks are useful if you're only dealing with multiple boolean values, but I needed to support integers as well, so I needed an easy way to say:  "Ok, bit 0 will be setting A, bits 1-6 will be setting B and bits 7-16 will be setting C."  So I created a little library called BitBit.

BitBit works by letting you create a new BitBit object with a schema that defines how a JavaScript object maps to certain bits.  For example, if you had a thermostat object with settings about a thermostat that you wanted to pack down into under 2 bytes, you could set up a new BitBit object like so:


What this is doing is creating a new BitBit object with a valid schema.  A schema must be an object with string keys that map to an array of 1 or 2 numbers.  A single number means it's a boolean and uses a single field.  2 numbers mean that its mapped to a number that uses the bits spanning the numbers given in the array (inclusive).  If you wanted to return a number that was just 0 or 1, you could use an array with 2 numbers that were the same.

Once you have built the schema, you can pack the settings down into an integer for storage.


As you can see above, once you pack the object, it will ignore anything not specified in the schema, which means you lose that data once it's unpacked.  A good way to use the unpacked data is to merge it back over your original object with lodash.merge.  This works well if the device you're sending it to and from can modify it.

In addition, you can use lodash accessor keys to get nested objects, array indices, etc.  There is an example of that on the Github Readme.

Well, that's about it, it's a small library I created since I needed something like that and didn't see anything out there that existed yet that satisfied those requirements.

If you like it feel free to contribute!  That's all for now, hopefully I'll have more time for these little side projects.

Wednesday, June 8, 2016

Card Creator V2 - A smoother workflow

I had a couple goals in mind when I set out to rebuild the Card Creator.  First, I wanted to be able to update a Google Spreadsheet document, and have the card creator read directly from there for building the cards, and second, I wanted to remove the Cairo dependency, because it wasn't working very well and I wanted to use Node4 which broke node-canvas.

Version 2 of the Card Creator does exactly that.  I hooked up the Card Creator into Google Sheets using the npm module: google-spreadsheet, and I swapped out node-canvas and Cairo for LibGD and node-gd.  What that entailed was a full rebuild of the card creation logic, but it was good to go back over it and update it and smooth out some of the minor issues.

In addition I redrew much of the art for the icons so that I could use non-pixel art instead, which scales better.  Granted, I am no artist, but I am doing all of the temporary art until I can find an artist to work with, if I decide to pursue this project seriously.

If you haven't read my previous post.  Card Creator is an application written in NodeJS that automates creating playing cards for a card game I'm working on.  It allows me to rapidly prototype and handles design, combining art assets, etc into print sheets for ease of production.

This new Card Creator program has improved my workflow significantly.  With the old program, making card changes required me to update the JSON files individually and change the values for each key/value pair in order to generate the cards.  Now I simply create a new row in my google document and run the program and it pulls in all the changes.  It's really nice for rapid prototyping.  I'm able to tweak values, update the descriptions, and then immediately run the program to generate the cards, and the sheetify script to create print sheets.


The script has a few points of interest you'll want to consider when using it.  First it's run by using the command: node creator-google-sheets.js which is the primary file.  (Make sure you're on branch version2).  The first thing you need to do to run it is include a google spreadsheet key to a google document.  In order to use the sheet, you'll need to publish it to the web, and then you'll have a link that has a long string in it.  That's your spreadsheet key.

I put my key in a private.js file which isn't included in the repository, so you can do the same, or you can simply replace the require('./private.js').google_sheets_key; on line 15 of the script with the string.  In addition if your spreadsheet has differently named headers for your spreadsheet, you'll need to update those in the mapping function (convertRow) on line 73.  That basically maps the row data to fields that the icon creator, background creator, etc are expecting.

The Card Creator works similar to before, it draws the background, the title, the description and the icons and then outputs it as a png file.  One thing to note, is that if you're working with node-gd, the documentation isn't fantastic, so it might take some trial and error to figure out how the functions work exactly.  For example, it took me a little bit to figure out which copy function worked for copying one image onto another.  Use copyResampled.  At least that's the only one that worked well for what I was doing.

I'm also rebranding the Card game I'm working on a bit, tossing some names around trying to find one I like.  I have a couple in mind, so we'll see if they stick in play testing.

Anyway feel free to check out the new Card Creator on my Github Page, if you're working with NodeJS linking up to google sheets documents is really quite simple and fantastic for automating work.  You can make the document read only (like I did) or allow programs to write to cells as well.  Nifty!

Cheers and thanks for reading!
Jason C / WakeskaterX


Tuesday, November 24, 2015

NodeJS Basics - Object Patterns & Differences

Ever wondered when you should use a Singleton vs. using an object instantiated for just that script or even just for that call?  Not knowing the scope of the object can cause all sorts of issues in a NodeJS application if you don't carefully consider when it gets instantiated and what is allowed to access it.

I'll quickly walk through a couple scenarios which look very similar, but have a much different impact on your code.  This may be obvious to you if you're a long time NodeJS developer, but it's something that has tripped me up a few times in the past.

First, let's take a look at the Singleton model and how it works.

The Singleton Model

In the NodeJS Handbook, Fred K. Schott introduces the NodeJS Singleton:
"In most languages, sharing an object across your entire application can be a complex process. These Singletons are usually quite complex, and require some advanced modifications to get working. In Node, however, this type of object is the default. Every module that you require is shared across your application, so there’s no need for any special classes or extra code."

In NodeJS we can very quickly and easily create singletons, but it's important to be careful of when to use them, so you don't get yourself in trouble.

Let's take a look at a sample function, which will be our singleton model using a basic Object Oriented Approach.  There are other ways to write Singletons as well, but I'm writing the Singleton this way to focus on the small differences between Singletons and Instantiated Objects and how to not get tripped up by them.


Above we have a basic function: MyClass, with a single attribute: name, and single function on that class: askName.  Once it's added to the module.exports it is globally available for any of your scripts to require.  The important thing to note here is that you are creating a new instantiated object that gets put on the global modules list:  new MyClass("testy1"). Any script that accesses it references the instantiated object.  So in a second script, you could access it like so:


And if you had a third script running in the application you could run it there as well:


Now this is the important thing to remember about Singletons, and the reason you use this pattern:  If you change the name in the first script on the same NodeJS process, it will change it in the second script as well.  So the following test script which runs both scripts, will output "Hello, my name is Bob" twice:


If you swap the order of these scripts, it will output "testy1" as the name and then "Bob".  So just make sure you remember that when you modify values in a Singleton, that it will have that effect on anything else that is accessing it.

When to Use

The Singleton pattern is particularly useful whenever you need to have something store values across all scripts.  Think of it as being "super global" to the node process.  But that also means that you shouldn't use it if you need to change things between web requests or want to have the object do different things in different areas of your process.

With this pattern, you don't need to create the objects every time.  When you require it, you're accessing the created and instantiated object, so you can use it's methods and values right away.

The Object Oriented Model

So what if you don't want to use the Singleton, or it doesn't suit your needs?  Well instead of instantiating the object on the module.exports, you can simply pass the constructor back and let the calling script instantiate it.

This fits more along the lines of traditional object oriented programming.  Rather than one single global object, you allow the calling script to create each object as needed.


The above script is exactly the same as the Singleton model with the exception of line 10.  Instead of passing in a new MyClass with a set name, we simply pass in the constructor, the function MyClass.  Then in our scripts, when we require it, we must instantiate it, creating the objects with their own values.


There are 2 ways I normally go about instantiating an object from a module.  The first is to simply require the class, as seen on line 2, and then instantiate it (line 3) separately or when you need it.  Or, if you know that you only need to instantiate it once at the start of the script, say in the case where you have an error library or a debugging library that has settings specific to an individual script, you can instantiate the require, as seen on line 5.

When to Use

The Object Oriented pattern is useful any time you want something to be local to your script or function.  One thing to watch out for is the scope of the object and where and when you instantiate a new one.  If you're going to be using something for a certain web request, make sure you instantiate it during that web request within the function.

One recent 'doh' moment I had was when I instantiated my object globally, but was using it in web requests that came in.  So while Node was handling multiple requests, the logic the object was doing was being accessed by 2 different processes and was being corrupted during the request.  Moving the instantiation into function where I handled the request solved the issue.

Looking back it seems like a simple mistake, but it threw me for a loop for a while trying to figure out how the data was being corrupted.

There are plenty of other patterns as well, but I wanted to focus on the very small change between a Singleton and a Constructor based pattern that can make all the world of difference in your NodeJS code if you don't realize how it's different.

Hopefully reading this saves you a headache in the future!  Thanks and if you enjoyed this article, please follow me on Twitter & Subscribe to the blog!  Cheers!

Tuesday, September 1, 2015

League of Legends API Challenge Entry - PowerSpike

On Monday, August 31st, I could enter no more code.  It was done, complete, or as much as it could be with the time invested.  I had finished my entry for the Riot Games, API Challenge #2.  I had created and submitted PowerSpike.

The API Challenge I took part in was to create a web application that referenced data sets from before and after a massive itemization change that Riot made to their game, League of Legends.  When I read the requirements, it sparked an idea.  What DID the data look like for champions as they got kills throughout the game?  How many kills in each minute of a game, does a particular champion get on average, and how did those item changes affect that?

And so I set out to satisfy that curiosity.  I began work on an application called PowerSpike, using NodeJS to serve up a web server, as well as run the parser which would collect the data.

Those two parts contain much of the application code.  

The Riot API Parser

First, the parser runs through each match, pulling down the data from the API and reading in the kill data for various champions in each match.  I chose to use the four data sets for the North American data, since those are the servers I play on and I wanted to limit the amount of data to specific regions.

When the parser starts up, it checks each of the matches in the riot test data files against a list of match IDs that have been parsed already and if they haven't been parsed already, it queues them up to be run.  Then it gets the data, grabs the kill times for each champion and stores that data in MongoDB as kills per each minute.

It does this slowly, waiting between each request so we don't hit rate limits by the API.

The Web Server & Client

The other half of the application is the web server which exposes a few endpoints for data retrieval from MongoDB, as well as the client static pages, which display the data in line charts over time throughout a game.

Chart.JS is the library for displaying data that I used and it's pretty good at the basics.  Each of the four data sets gets compared to one another.  I also modified the chart to allow the user to toggle datasets with the legend as well, since the basic UI can be cluttered.

There are some very interesting data points for a few champions as well.  For example, Ahri gained a significant boost in early game kills for normals after the patch.  This could be due to the much earlier Needlessly Large Rod that players could now purchase at an earlier time.

But either way, the data points are pretty interesting. and there are a few champions like Kayle and Teemo that have some pretty interesting data to look at.

Hosting

I decided to host the project on a web server, so I spun up a new Digital Ocean droplet and purchased powerspike.xyz at NameCheap.com for 1 dollar for the first year.  I highly recommend Digital Ocean as I run many of my projects off their boxes.  You get full control of the VPS and it's great for being able to do all sorts of interesting projects.


It was a pretty fun challenge, and it's now in review for the next month, so we'll see how it goes!

You can view the application at:


And the full source code is at my GitHub:


Feel free to check it out and let me know what you think!

Cheers,
Jason C.

Monday, June 22, 2015

Article Curator - Web Scraping with Casper JS and Node.

Article Curator

I was on vacation recently and had some time to start up a side project, and so I created a custom article curator for a website I'm working called The Tamriel Underground.  I wanted to automatically scrape sites and pull in articles that I could link to and show on my site, so that I could have automatic content populate the articles section.

And it works fairly well for the initial build.  I used CasperJS for the web scraper and NodeJS to spin up a web service which would save the data to a Postgres database.

You can view the whole project on my Github.  It's just 2 JavaScript files, one for the Scraper and one for the Web Server.  It's a fairly small project at the moment, but I plan to build a few more scrapers for other sites to pull in data.

The Code

Both files are only about 200 lines of code combined, so it's not a ton of code to walk through, but this was my first time using Casper and Phantom so it took me a bit of work for those 200 lines.

The web server gets run constantly and stays up using node forever, and the web scraper is run as a cron job every 2 hours.

First let's take a look at:

eso_pnote_curator.js

The goal of this script is to access the Elder Scrolls Online Website, log in through the Age Gate, and then scrape the Patch Notes site for the links to the patch notes. The script then iterates through those links to pull down the html used in the article.

At the top of the file are the declarations:


Here, I specified the urls that I wanted to access, initialized Casper and created an array and object to store the data as well as set a few settings.  Pretty straight forward nothing special here.

Next, lets skip a few lines down to line 87.


Here is the starting point for Casper.  The start function opens up the age_gate_url which points to the page needed to be filled out before going any further, then it waits for the page to load by waiting for the selector to show up, and then fills out the form.

Then it opens up the page where the patch note links are and calls out to our getPatchNoteLinks function which we'll look at later, and evaluates that within the page.

Finally the script runs Casper.  This is what actually makes all the steps we've declared so far happen.  And once all those steps are done and the script has all of the patch notes links, it makes the call to getNotes, which is the callback after Casper is finished running.

Now the script can start parsing those links it's accumulated.


getNotes iterates through each link, calling Casper.run at the end of the link processing, recursively calling itself at the end of each run, until all links are processed, in which it calls out to the save function.


Above you can see the three functions we've seen used so far.  The first one, which was run in the evaluate statement is the middle one, getPatchNoteLinks.  Now this function is run inside the loaded page.  It's just like client side javascript or if you ran it in your console.  And it will return the value back to our PhantomJS/CasperJS 'scope'.

You can see the evaluate statement used again below in the curateLink function as well.  Here it is taking a link, using that evaluate to grab the title from the page and returning it, grabs the entire article which is between the article tags, and then adds all of that information to the patch_note_info object.

And finally, once all this is done, it calls out to the final function, saveInfo.

The reason that there are two parts to this application is that CasperJS and PhantomJS are NOT node applications, and thus there are no server side database drivers (that I know of) built for them for Postgres.  However we CAN post that data to a separate NodeJS server that will store it for us.

saveInfo does exactly that.  We post our stringified object with all of our data to a NodeJS web server and let it do the saving for us.

This also allowed me to keep the curator light weight.  I have a single location that handles saving to the database, and then curators can simply handle the scraping and POST it when done.

So let's look now, at the simple web server.

article_server.js

Again, there are basic declarations and requires at the top:

For this app I used express, simply because I'm familiar with it and it's easy to get a simple REST server up and running, however there are much lighter weight options out there for something small like this.

The script requires:

  • express - a nice REST framework
  • pg - this is the Postgres connection module allowing access to the database
  • body-parser - allows easy parsing of JSON data
  • async - a nice framework for iterating through database calls with different datasets.
And then the script sets the connection string and starts the app.



Then comes the bulk of the web server.

First it includes the body parser so that it can parse the data sent easily.

Then it has the one endpoint POST /save.  Here it grabs the articles, pulls out the sites and pushes the data to an array.  Then it connects to the database, and checks if those sites already exist in the database.  If the sites exist, it skips them, otherwise adds the data that it doesn't have yet to the database.


And that is exactly what these two functions do.  insertSites will takes the sites the database don't have already stored, and iterates through each one with the async library.  This is where the async library is convenient.  It allows easy iteration through the asynchronous methods which are needed for the database insertion.

And that's all there is to the Curator!

That data then gets consumed by the Tamriel Underground, which is a Django application and converted to a model.  I lined up the model schema in python with the way I'm entering the data and it works quite well.

The articles get added to the page, and you can view them with the plus button.



Right now I'm using the html I get from Elder Scrolls Online as is, but I'm going to create a cleaning function to strip out anything that might be invalid.  Since I'm going to be scraping the html and displaying it on my site, I want to make sure that no one can insert any nasty script tags or links that might cause a security concern.

Since I know the source that I'm scraping I'm not terribly concerned at the moment, but as I add more sources to the curator, I'll definitely want a robust input cleansing module.

So that's pretty much it, feel free to pull the Github repo down and play around with it and let me know what you think.

Cheers,
Jason C.

Thursday, March 5, 2015

NodeJS - A Little Experiment in Load Testing and Clustering

Load Testing NodeJS on Multiple Cores

Check Out the Git Hub project HERE

Something I wanted to investigate was using Node JS on a multi core system to do CPU intensive applications under high load, so I created a project to do just that.

There are a few ways to use multiple cores in NodeJS, two of which are Cluster (part of the NodeJS API) and WebWorker Threads.

I will preface this by saying I am no NodeJS expert, this project was simply for learning on my part and there may be a better way to go about this with an NGinx set up, but I wanted to share my findings and maybe you'll find it cool too.

The Planning Stages

I started the project out with a simple NodeJS and Express server, which when called with a number in the query, would calculate and return the Fibonacci sequence for that number to simulate high CPU calculations.

My first thought was to use Web Worker Threads to spin up a thread to do work on, and just pass in the data and let it work on a background thread, but this caused issues under high load as too many threads were being spun up (I had no cap at the time) and at the start they were descoping and causing segmentation faults.

I ended up fixing the Seg Fault issue, but even so, they were burning through all of my VMs 2GB of memory and crashing.  

I attempted to create a Worker Pool function to control the amount of threads at any one time, but it quickly got quite complicated.  The idea was to use a queue and simply drop information off and process when ready, but with web requests and waiting on data, this as well got very complicated.

Then I stumbled across the Cluster section of the NodeJS API and found my solution.  With the Cluster API you can spin up multiple instances of your node server on the same port.  So I decided to test if this truly would improve my performance.

Load Testing

I installed loadtest in order to deliver high concurrency testing to my application and began to test the application.  On one branch of my git project I had the vanilla, single threaded NodeJS server which would generate a Fibonacci sequence, and on my other branch I had my project which generated 4 servers on a single port, which would use up the multiple cores.

This was a MUCH simpler solution to building a multi-threaded system within a single event loop and was extremely easy to set up.

So the load testing began.  I called out to my application like so:

loadtest -t 20 -c 32 http://localhost:3030/fib?num=25

I pinged my application using various concurrency values for 20 seconds testing periods.  I tested up from 1 concurrent connection up to 1000 connections.

While I don't believe all four cores were actually being used (as it was on a VM and my computer didn't crash) I did see quite an improvement using the multiple core approach (as one would expect).

The Results

The results from the load test are as follows.  Load test outputs the maximum response times for 50%, 90%, 95%, 99%, and the maximum response time as well as Requests Per Second, Mean response time, and Total Requests.

Computer Specs:  The VM was an Ubuntu 14 VM with 2 GB Ram, 4 processors.

Each request was made with a Fibonacci sequence of 25 for 20 seconds.

NodeJS Single Event Loop

ConcurrencyCompleted Requests50%(ms)90%(ms)95%(ms)99%(ms)Max (ms)Req / SecMean Lat (ms)
648891135169180245286429150
20081103361276135833007364405480
1000817812353726762615726162564092510

NodeJS Multi-Core with Cluster

ConcurrencyCompleted Requests50%(ms)90%(ms)95%(ms)99%(ms)Max (ms)Req / SecMean Lat (ms)
64150097015519332756075080
200144982524765677561062596330
100014923117219542538361876417181350

As you can see the results from using the multi-core approach were much better than the single event loop, and using Cluster is super easy to do.  

You can fork the GitHub project here:  https://github.com/WakeskaterX/NodeThreading

The main fork is (well master too, but) hostedVM which has the standard deployment of nodeJS and express, and the multi-core approach is on the hostedVM_multicore branch.

Feel free to test it as well and let me know how your results are with better machines than my very low powered VM.

Cheers,
WakeskaterX