Running R in an Azure C# WebJob

I had the task to run R in an C# azure web job, I could of used the Azure Machine Learning Studio in order to run the R script, and executed this process via a web call. But I wanted the ability to potentially host this app outside of Azure, an a simple C# web app would allow me that with simple modifications.

This was my first time looking at R, so I had to spend quite a bit of time in R Studio to get to grips with the basics of running the scripts and handling packages. The next step was to decide on how to run the script, after a bit of searching I found the RDotNet nuget package which promised to provide a simple way of running pieces of R code within .net.

The simplest example I got running to prove the concept:

This expects a standard installation of R on your windows machine, I believe it can handle installations on unix and mac if running in mono. This script ran perfectly, I was able to get this running nicely in Linqpad as well. The next issue faced was how to get this running if you dont have a standard installation of R, as would be the case with the Azure web app.

The SetEnviromentVariables call accepts to parameters, rPath and rHome, so if your installation was else where, or even running from a usb stick or network drive you could still use RDotNet. With some lots of searching on StackOverflow, I finally found what paths the SetEnvironmentVariables call was expecting. I copied my installation of R to a new dir and updated my linqpad script with the following:

The script continued to run successfully, so with the concept proved I could move onto creating the Azure web job. Using the Kudu tools website (https://<WEB_APP_NAME>.scm.azurewebsites.net/DebugConsole), I copied the R-3.3.2 folder and its contents to /site/wwwroot/,

Web Job Folder Structure

Web Job Folder Structure

and updated the rPath and rHome variables to these new locations:Pushing the new web app and letting it run, provided the same output ’10′ in the azure streaming logs for the app. Success!

Packages

Soon after getting the basic R script running I ran into issues adding new packages, the basic error boiled down to not having rights access to download or install new packages. (There may be a simple way round this, such as changing where R is located on the web app, or some config settings).

The work around involved downloading the package I was looking to use, in this case RODBC, and uploading the zip to the web app, you can see my packages folder in the screenshot above. Next you have to install this package in your R script, I achieved this with the following:

National Three Peaks

I thought it was about time to write something down about our National Three Peaks Challenge that we complete in May 2016. After climbing each of the peaks individually over the previous 6 months we decided it would be worth attempting to complete them all within 24 hours.

Me, Peter and Sam

Three Peaks Challenge (me, Peter and Sam)

We spent a bit of time trying to work out the best way to climb the 3, the common convention is to tackle them from north to south, but there is differing opinions on when to start the climbing. One of the options is to start climbing Ben Nevis in the early evening, allowing you to get down the mountain before total darkness, then travel onto Scafell during the night, starting the 2nd climb at day break. The downside we saw with this plan was the travel from Scafell to Snowdon, which would be around 7-9 am, perfect for hitting bad traffic around Manchester. We already knew we were going to be cutting it fine with all the travel and none of us wanted us to fail the challenge because of bad traffic. So the 2nd option was the start Ben Nevis at around 9 am, which means starting the Scafell climb at around 8-9 pm if traffic was kind (it wasn’t), which also means doing Scafell in the dark, but also means we miss any morning traffic heading to Snowdon.

Me & @Bonxy decided it would be worth doing Scafell in the dark in march, as a bit of practice to see if it was a viable option. Although the weather was dry it ended up being VERY windy and foggy at the summit, which made finding the route to the top difficult to say the least. Visibility was about 1-15 meters,  which was too short to be able to see the next cairn, luckily we had a handheld GPS which we followed to the top, but was a bit hairy. I think we would have had to turn back if we hadn’t had one, as there was no way of knowing which direction to go. To illustrate this, when setting off back down we knew the rough direction to head, but after 20 seconds we checked the GPS and found we had already veered quite a way of course and had no idea. Comparing our first attempt in the day with this night climb, it took us about another 30 minutes to complete it in the dark, almost all of the time was wasted on the summit trying to see through the fog.

Although not all plain sailing we decided that doing Scafell at night wouldn’t be too difficult, especially as it would be the 3rd time of climbing it in 6 months, and hopefully in May the weather would be a bit kinder.

The plan was simple, we’d rent a 9 seater minibus, ideally we’d have gone for 12 seats to give us more space but there were none available for our dates, which would give us lots of room for kit and hopefully some sleep. We needed a driver for the trip, luckily Jim stepped in for us, who probably single handedly kept us on track with his efficient driving. After setting off from Teesside we would arrive in Perth for tea and would allow us an early start at Ben Nevis, getting up at 6am. From there its only an hour or so drive to Forth William and the start of the challenge.

At 9am we started the challenge, cheered on by Jim and @Becks, the weather was kind to us, with it being fairly mild and dry, although from the start we knew there was heavy cloud cover from about the halfway mark. Which would mean this was the 2nd time we’d summit Ben Nevis with no visibility, a tad annoying.

We made good time, reaching the peak at xx, and as predicted covered in cloud and snow. The snow is always a massive pain when reaching the top, but makes for a fun and quick descent, as long as you don’t fall or lose your walking poles. Halfway down with xx on the clock we were making good time, and we all felt pretty good, we bumped into another group attempting the challenge on their way up, we would end up bumping into them again at the next climb.

Ben Nevis Stats

Ben Nevis was completed in 4hrs 50mins, much faster than me and Sam were expecting and made up very optimistic for the 24 hour limit. Unfortunately on the last mile or so the heavens opened and we all got a bit wet, luckily we had a 5 hour drive to Scafell in order to try and get everything dry. Now for the best tip I would give anyone attempting the challenge is try and do it in a minibus, the 3 of us ended up all sat in the back row with the middle rows seats lowered allowing us to all sit in comfort with our legs outstretched.

All was going well until we hit the outskirts of Glasgow, and the inevitable traffic, we lots around 30 mins in this traffic, which wasn’t too bad, to make matters worse, soon after turning off the M6 at Carlisle we hit traffic and several road closures, we started getting a bit worried, we’d lost over an hour simply due to traffic and closed roads. We made it to the Wasdale car park for around 8pm, weather was being kind to us again and we all still felt pretty good.

We knew Scafell Pike quite well, having done it twice before in previous months, but in my opinion is one of the harder initial climbs, the accent starts straight away and only gets worse until you finally scramble up the rocks for the final hike to the summit. The short scramble is even worse at night in the wind, and to be honest probably isnt the best route, but I’ve always round it quite fun and breaks up the monotony of the hike. After xx hours we made it to the summit in the dark, fog and wind, again we were making good time. The trip back down was uneventful, the optimism grew as we knew we only had one more mountain to go, we did notice a group of walkers that seemed to be coming down from Scafell straight off the side of the mountain, we watched the lights slowly moving down as we also made our decent.

Shortly before the car park we bumped into a large group of lads also doing the challenge, it turned out they were the source of the lights and had in fact got totally lost, misread their maps, and just had to simply walk straight down in the dark and hope they met up with the trail. This is where we were very grateful to have our GPS unit to prevent the same happening to us.

Scafell Pike was completed in 3hrs 30mins, bringing our challenge time to 15 hours, that meant 9 hours to get to Snowdon and complete the climb, it seemed quite possible. A change of clothes, some food and water in us we set off at around 12am, we quickly realised something wasn’t quite right with the minibus, turns out we had a flat tyre. So there we were pitch darkness, 5 of us running round the minibus trying to find the spare tyre, the jack and the locking wheel nut. The wheel was finally changed after about 20 minutes, but we were now even more tired, covered in mud and oil and a bit pissed off. The icing on the cake was yet more road closures in the lakes, the chance of finishing the challenge in time was seeming less and less likely, it was better to try and get some kip and hope for the best.

Puncture

Somehow we got Pen-y-Pass at around 4:45am, giving us 4  hours to complete the hike, which normally would be very possible, but we were tired and feeling a bit weak, well 2 of us were, Peter (who was doing this challenge for some training before a 54 mile ‘run’ later in the year) was looking like he’d just started the challenge. The Snowdon climb was very uneventful, there wasn’t many people about, and the 3 of us weren’t that chatty, we made it to the summit 2 hours in, meaning we had 2 hours todo the decent, which looked quite achievable, I was starting to feel very tired and weak by this point, I realised I hadn’t had enough to eat after Scafell and with struggling to sleep in the minibus, not had enough sleep at all.

Snowdon Summit

We did however make it down with 31 minutes spare, which with the road closures, traffic and flat tyre I think was a pretty good time. After having some sleep and food I started to feel pretty good, in fact by the next day I was surprised at how good I felt, I was expecting to ache for the following week.

Now just the Yorkshire Three Peaks to complete next.

Gallery: youtube

Video: flickr

Ben Nevis: route

Scafell Pike: route

Snowdon: route

Ben Nevis [1/3]

Me and @Bonxy have just got back from climbing Britain’s highest mountain, Ben Nevis. With little to no training we decided to tackle the biggest out of the 3 peaks, which probably wasn’t the best idea.

Me & Sam

Me & Sam at the summit

We drove up Friday afternoon, and got to Fort William for around 8pm, took about 5hrs 30. At around 6am we got up and started preparing for the climb, but the midges in Scotland like to get up early as well, so before we got bitten to death we set off.

It wasn’t the best of days for the weather, there was lots of clouds obscuring the mountains around us, which didn’t bode well for us getting some good views at the top. We made the quick walk to the Glen Nevis Visitor Centre and crossed the footbridge to start the climb, by this point I was already too warm and starting to sweat.

Me and @Bonxy decided we would try and stop every 40 mins or so to catch our breath and take on some water, which worked out quite well, after around an hour we were in the clouds, and getting rained on, but I still wasn’t putting my jacket on due to how much I was sweating, we made it to the half way point with no real problems.

Halfway Point

Me at the halfway point

One of the worries I had before the walk was about getting lost as we got up higher, but I needn’t have worried as there was two different charity events going on, plus plenty of people doing the three peak challenge, so there was a constant stream of people going up and down, plus the occasional traffic jam on the narrower sections.

Soon after we hit the first of 8 switchbacks, these seemed to go on and on, but we could take a quick 2 min break at the top of each. On the final couple of switchbacks we started to see the snow, which at first was a welcome site, as it meant we were near the top, but with the lack of walking poles the going was tough as we kept slipping on the fresh snow and ice.

Finally we reach the top and saw the crowd of people hiding from the wind around the stone shelter. Unfortunately due to the bad weather there wasnt much to look at, we had no reward from our hard walk apart from our quick lunch. A few picture later we were on our way back down.

Route

For me the walk down was worse, two and a half hours of trekking down the hill killed my knees, so I was very happy when we finally left the clouds behind and could see the valley floor. We decided to take the more direct route down which brings you out at the hostel, which might not of been the best idea with my dodgy knees, but it did get us down quicker.

The whole walk took us around 6hrs 30, which I was quite happy with, we covered 16km in total, and my fitbit reckoned I took 35,000 steps and burnt around 5500Kcal.

Elevation

Next on the list are Sca Fell and Snowden to complete the three peaks.

River Level Twitter Bot

With all the bad weather, and with moving to opposite a river I had been keeping an eye on the river on my way to work and found the Environment Agency site has real time (sort of) data for most of the rivers in the UK.

So I decided to knock up a quick script to scrape the page for the river level and tweet to a newly created account.

Link: @RiverAireLeeds

Windows 8 – Desktop on start up

Update: I thought it’d be easier to just give you the .reg file to download that will quickly update your system, just download and double click the .reg file. You can open it up in notepad and see what the file is going todo, simply changing the Shell key to the new value. :)

I’ve just switched to Windows 8 at work, and its taking some getting used to. One annoying feature is that when you log in the metro ui is shown first rather than the desktop, now since i spend 99% of my time on the desktop, I want to see that first.

After a bit of googling I found a solution, without having to install 3rd party apps. Just a quick registry edit.

Open up RegEdit and head to…

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\Current Version\Winlogin

Then find the ‘Shell’ key.

By default, it is set to “explorer.exe”.

Change the value to “explorer.exe /select,explorer.exe”

PhantomJS Xbox Achievements

I originally tracked my xbox achievements by using macros and batch scripts, this allowed me to grab my own achievement data, and store it in my db for viewing on the web. This worked great apart from the inherent problem of have macros physically opening and closing browsers every 15 minutes of my virtual machine, every day or so the scripts would break or Firefox would freeze, so I was always having to keep on top of it, not ideal.

A few months later I came across XboxAPI.com, this handy (FREE) site wraps up all the xbox.com functionality and provides users with a simple to use api to get game info, friends, achievements and other data. So I scrapped my macros and jumped on this api, this worked spot on and still does. The only downside is that due to constraints on the Xbox.com website the achievement images were only available in black and white and this didn’t look to great on my website.

So my next plan was to use PhantomJS, after a few weeks of playing around with it on and off I didn’t get too far, then I came across this script on github, after a quick test, it did indeed log me into Xbox.com and grab my friends page. So a few hours later and some tweaks I’ve got it to get the users latest game achievement data in JSON, its not polished and could break at any time if Microsoft change the Xbox.com website, so bear that in mind. But hopefully it’ll give someone a good place to jump off from. :)

/*
This script provides a quick and dirty way of getting your most recent game achievement json data.

This is a lightly modified version of Jabbslad's script found here: https://gist.github.com/1653745

James Coverdale - imjam.es 2012

*/

var fs = require('fs'),
    system = require('system');

var name = "";

if (phantom.args.length !== 2) {
    console.log('Usage xbox.js <username> <password>');
    phantom.exit();
}

var page = new WebPage({
    'page.settings.loadImages': false
});

/*
* Callback to process page when finshed loading
*/
page.onLoadFinished = function(status) {
    if (status !== "success") {
       console.log("Unable to access network");
    } else {
        var url = getPageUrl();

        switch (true) {
        case /login.srf/.test(url):
            // Step 1 - Login
            page.evaluate(fillFormFunctionAsString());
            break;
        case /post.srf/.test(url):
            // Step 2 - Process Cookies
            break;
	case /Details/.test(url):
	    // We got the achievements
            parseGame();
            phantom.exit();
            break;
        case /Activity$/.test(url):
            // We did it!
            parseFriends();
            break;
        default:
            // uh oh we hit an unexpected url
            phantom.exit(1);
        }
    }
}

function getPageUrl() {
    return page.evaluate(function() {
        return location.href
    });
}

//This whole function will need reworking ideally, must be a better way of grabbing the json
//but this works for now
function parseGame() {

	var index = page.content.indexOf("broker.publish(routes.activity.details.load, ") + 45;
	var lastindex = page.content.indexOf("</script>", index);

	var json = page.content.substring(index, lastindex);
	json = json.split("});").join("");
	json = json + "}";

	// You could just push the json to the console to allow use in the command line, i.e. pipe it to another script.
	f = null;
	f = fs.open(name + '.json', "w");
	f.write(json);
	f.close();

}

function parseFriends() {

   try {

	   var contents = page.content.substring(page.content.indexOf("<div class=\"activityPage activity\">"), page.content.indexOf("</ol>  <div class=\"clearfix\"></div></div>"));

	   // Again this string manip is not ideal, too much that could go wrong if the site changes
	   var index = contents.indexOf("<a href=\"/en-US/Activity/Details?titleId=") + 41;
	   var url = contents.substring(index, contents.indexOf("\">", index));
	   var nameindex = contents.indexOf("alt=\"", index) + 5;
	   name = contents.substring(nameindex, contents.indexOf("\"", nameindex));

	   // make the game name lower case and remove any spaces
	   name = name.toLowerCase().split(" ").join("");

	   // Load the latest game page
	   page.open(encodeURI('https://live.xbox.com/en-US/Activity/Details?titleId=' + url));
    }
    catch (e) {
       console.log(e);
    }
}

/*
* Hack due to phantomjs page.evaluate limitations:-
* http://code.google.com/p/phantomjs/issues/detail?id=132
*/
function fillFormFunctionAsString() {
    return "function() {"
         + "var form = document.querySelector(\"form[name='f1']\"); "
         + "var login = form.querySelector(\"input[name='login']\"); "
         + "login.value = '" + phantom.args[0] + "'; "
         + "var passwd = form.querySelector(\"input[name='passwd']\"); "
         + "passwd.value = '" + phantom.args[1] + "'; "
         + "var kmsi = form.querySelector(\"input[name='KMSI']\"); "
         + "kmsi.value = '2'; "
         + "form.querySelector('#idSIButton9').click();}";
};

page.open(encodeURI('https://live.xbox.com/en-US/Activity'));

The script logs in, gets the game ID of the last game played (or currently being played), then heads to that game page for the user and pulls the JSON from the page and saves it as a .json file. You could modify it to not save the JSON and just output it to the console if you want, but in my setup I needed the .json file to upload to my webserver in order to do extra processing upon it.

To run the script simple head to PhantomJS.org and download the relevant version, runs on Windows, Mac and Linux. then simple from the command line call…

phantomjs XboxScript.js username password

Simple, you will need to edit the name of the .js file to what you have saved it, and will probably need to point to the phantomjs executable on your system.