Archive for the Code Category

Running R in an Azure C# WebJob

I had the task to run R in an C# azure web job, I could of used the Azure Machine Learning Studio in order to run the R script, and executed this process via a web call. But I wanted the ability to potentially host this app outside of Azure, an a simple C# web app would allow me that with simple modifications.

This was my first time looking at R, so I had to spend quite a bit of time in R Studio to get to grips with the basics of running the scripts and handling packages. The next step was to decide on how to run the script, after a bit of searching I found the RDotNet nuget package which promised to provide a simple way of running pieces of R code within .net.

The simplest example I got running to prove the concept:

This expects a standard installation of R on your windows machine, I believe it can handle installations on unix and mac if running in mono. This script ran perfectly, I was able to get this running nicely in Linqpad as well. The next issue faced was how to get this running if you dont have a standard installation of R, as would be the case with the Azure web app.

The SetEnviromentVariables call accepts to parameters, rPath and rHome, so if your installation was else where, or even running from a usb stick or network drive you could still use RDotNet. With some lots of searching on StackOverflow, I finally found what paths the SetEnvironmentVariables call was expecting. I copied my installation of R to a new dir and updated my linqpad script with the following:

The script continued to run successfully, so with the concept proved I could move onto creating the Azure web job. Using the Kudu tools website (https://<WEB_APP_NAME>.scm.azurewebsites.net/DebugConsole), I copied the R-3.3.2 folder and its contents to /site/wwwroot/,

Web Job Folder Structure

Web Job Folder Structure

and updated the rPath and rHome variables to these new locations:Pushing the new web app and letting it run, provided the same output ’10′ in the azure streaming logs for the app. Success!

Packages

Soon after getting the basic R script running I ran into issues adding new packages, the basic error boiled down to not having rights access to download or install new packages. (There may be a simple way round this, such as changing where R is located on the web app, or some config settings).

The work around involved downloading the package I was looking to use, in this case RODBC, and uploading the zip to the web app, you can see my packages folder in the screenshot above. Next you have to install this package in your R script, I achieved this with the following:

River Level Twitter Bot

With all the bad weather, and with moving to opposite a river I had been keeping an eye on the river on my way to work and found the Environment Agency site has real time (sort of) data for most of the rivers in the UK.

So I decided to knock up a quick script to scrape the page for the river level and tweet to a newly created account.

Link: @RiverAireLeeds

Windows 8 – Desktop on start up

Update: I thought it’d be easier to just give you the .reg file to download that will quickly update your system, just download and double click the .reg file. You can open it up in notepad and see what the file is going todo, simply changing the Shell key to the new value. :)

I’ve just switched to Windows 8 at work, and its taking some getting used to. One annoying feature is that when you log in the metro ui is shown first rather than the desktop, now since i spend 99% of my time on the desktop, I want to see that first.

After a bit of googling I found a solution, without having to install 3rd party apps. Just a quick registry edit.

Open up RegEdit and head to…

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\Current Version\Winlogin

Then find the ‘Shell’ key.

By default, it is set to “explorer.exe”.

Change the value to “explorer.exe /select,explorer.exe”

PhantomJS Xbox Achievements

I originally tracked my xbox achievements by using macros and batch scripts, this allowed me to grab my own achievement data, and store it in my db for viewing on the web. This worked great apart from the inherent problem of have macros physically opening and closing browsers every 15 minutes of my virtual machine, every day or so the scripts would break or Firefox would freeze, so I was always having to keep on top of it, not ideal.

A few months later I came across XboxAPI.com, this handy (FREE) site wraps up all the xbox.com functionality and provides users with a simple to use api to get game info, friends, achievements and other data. So I scrapped my macros and jumped on this api, this worked spot on and still does. The only downside is that due to constraints on the Xbox.com website the achievement images were only available in black and white and this didn’t look to great on my website.

So my next plan was to use PhantomJS, after a few weeks of playing around with it on and off I didn’t get too far, then I came across this script on github, after a quick test, it did indeed log me into Xbox.com and grab my friends page. So a few hours later and some tweaks I’ve got it to get the users latest game achievement data in JSON, its not polished and could break at any time if Microsoft change the Xbox.com website, so bear that in mind. But hopefully it’ll give someone a good place to jump off from. :)

/*
This script provides a quick and dirty way of getting your most recent game achievement json data.

This is a lightly modified version of Jabbslad's script found here: https://gist.github.com/1653745

James Coverdale - imjam.es 2012

*/

var fs = require('fs'),
    system = require('system');

var name = "";

if (phantom.args.length !== 2) {
    console.log('Usage xbox.js <username> <password>');
    phantom.exit();
}

var page = new WebPage({
    'page.settings.loadImages': false
});

/*
* Callback to process page when finshed loading
*/
page.onLoadFinished = function(status) {
    if (status !== "success") {
       console.log("Unable to access network");
    } else {
        var url = getPageUrl();

        switch (true) {
        case /login.srf/.test(url):
            // Step 1 - Login
            page.evaluate(fillFormFunctionAsString());
            break;
        case /post.srf/.test(url):
            // Step 2 - Process Cookies
            break;
	case /Details/.test(url):
	    // We got the achievements
            parseGame();
            phantom.exit();
            break;
        case /Activity$/.test(url):
            // We did it!
            parseFriends();
            break;
        default:
            // uh oh we hit an unexpected url
            phantom.exit(1);
        }
    }
}

function getPageUrl() {
    return page.evaluate(function() {
        return location.href
    });
}

//This whole function will need reworking ideally, must be a better way of grabbing the json
//but this works for now
function parseGame() {

	var index = page.content.indexOf("broker.publish(routes.activity.details.load, ") + 45;
	var lastindex = page.content.indexOf("</script>", index);

	var json = page.content.substring(index, lastindex);
	json = json.split("});").join("");
	json = json + "}";

	// You could just push the json to the console to allow use in the command line, i.e. pipe it to another script.
	f = null;
	f = fs.open(name + '.json', "w");
	f.write(json);
	f.close();

}

function parseFriends() {

   try {

	   var contents = page.content.substring(page.content.indexOf("<div class=\"activityPage activity\">"), page.content.indexOf("</ol>  <div class=\"clearfix\"></div></div>"));

	   // Again this string manip is not ideal, too much that could go wrong if the site changes
	   var index = contents.indexOf("<a href=\"/en-US/Activity/Details?titleId=") + 41;
	   var url = contents.substring(index, contents.indexOf("\">", index));
	   var nameindex = contents.indexOf("alt=\"", index) + 5;
	   name = contents.substring(nameindex, contents.indexOf("\"", nameindex));

	   // make the game name lower case and remove any spaces
	   name = name.toLowerCase().split(" ").join("");

	   // Load the latest game page
	   page.open(encodeURI('https://live.xbox.com/en-US/Activity/Details?titleId=' + url));
    }
    catch (e) {
       console.log(e);
    }
}

/*
* Hack due to phantomjs page.evaluate limitations:-
* http://code.google.com/p/phantomjs/issues/detail?id=132
*/
function fillFormFunctionAsString() {
    return "function() {"
         + "var form = document.querySelector(\"form[name='f1']\"); "
         + "var login = form.querySelector(\"input[name='login']\"); "
         + "login.value = '" + phantom.args[0] + "'; "
         + "var passwd = form.querySelector(\"input[name='passwd']\"); "
         + "passwd.value = '" + phantom.args[1] + "'; "
         + "var kmsi = form.querySelector(\"input[name='KMSI']\"); "
         + "kmsi.value = '2'; "
         + "form.querySelector('#idSIButton9').click();}";
};

page.open(encodeURI('https://live.xbox.com/en-US/Activity'));

The script logs in, gets the game ID of the last game played (or currently being played), then heads to that game page for the user and pulls the JSON from the page and saves it as a .json file. You could modify it to not save the JSON and just output it to the console if you want, but in my setup I needed the .json file to upload to my webserver in order to do extra processing upon it.

To run the script simple head to PhantomJS.org and download the relevant version, runs on Windows, Mac and Linux. then simple from the command line call…

phantomjs XboxScript.js username password

Simple, you will need to edit the name of the .js file to what you have saved it, and will probably need to point to the phantomjs executable on your system.

Image Overlay CSS

My friend wanted a simple way of overlaying an image (or text) on an image, ideally using CSS. After a quick google I found a nice looking example of what he was after. Juts going to post it here so I have it for future reference.

null

HTML:

<div class="image">
 <img src="http://farm8.staticflickr.com/7151/6707334701_af50daf3c5.jpg"/>
 <h2><span>Roseberry Topping</span></h2>
</div>

 
CSS:

.image {
 position: relative;
 width: 100%; /* for IE 6 */
}

h2 {
 position: absolute;
 top: 200px;
 left: 0;
 width: 100%;
}

h2 span {
 color: white;
 font: bold 24px/45px Helvetica, Sans-Serif;
 letter-spacing: -1px;
 background: rgb(0, 0, 0); /* fallback color */
 background: rgba(0, 0, 0, 0.7);
 padding: 10px;
}

Demo: http://imjam.es/sam/test.html
Source: http://css-tricks.com/text-blocks-over-image/

Kindle – Unofficial Guardian Subscription

Having received a new 4th gen Kindle for christmas I signed up for the Guardian paper 2 week trial from Amazon. I’m quite impressed at how good the newspaper reading experience is like on the Kindle. The paper gets delivered everyday at around 6am, so when I get on the train, todays issue is sat at the top of my home page ready to be read. The subscription currently costs £10 per month, which isnt too bad, they charge £1 per issue if you don’t want to subscribe.

Guardian Kindle

I came across this blog post from Mark Longair, he has used the Guardian’s own open api to create a Kindle version of the paper, in fact he did this before the Guardian had their official Kindle version to download. Mark has the most recent edition on a page on his site, this is created ‘early’ each day. Now having compared the paid and this ‘free’ version, there isnt much difference, the majority of articles are there, including the magazines etc. I wanted a way to get this delivered to my Kindle every morning, much the same as my current paid subscription works.

Amazon allow you to email your kindle with attachments, which are then automatically downloaded to your device. So I set about creating a simple PHP script that will allow me to download the latest issue and email it to my kindle. Below is the code I used, I couldn’t get the attachment to work if the file was on a remote server, so the first thing the script does, is go and grab the todays issue from blah blahs website. Then the newly grabbed file is then attached to an email and sent to the specified email address.

Simply replace the To, From, Subject and Message variables with which ever values you require. Make sure to add the from address your going to use to your Amazon/Kindle account, to authorise this email address for delivery to your device.

<?php
 
 $filename = "gdn-".date("Y")."-".date("m")."-".date("d").".mobi"; 
  
 $handle = fopen("http://mythic-beasts.com/~mark/random/guardian-for-kindle/".$filename, "rb");
if($handle){
  $somecontent = stream_get_contents($handle);
  fclose($handle);

  $handle = fopen($filename, 'wb');
  if($handle){
    if (fwrite($handle, $somecontent) === FALSE) {       
       exit;
    }    
    fclose($handle);
  }else{     
     exit;
  }
}
 
// array with filenames to be sent as attachment
$files = array($filename);
 
// email fields: to, from, subject, and so on
$to = "hello@email.com";
$from = "me@me.com"; 
$subject ="subject"; 
$message = "message";
$headers = "From: $from";
 
// boundary 
$semi_rand = md5(time()); 
$mime_boundary = "==Multipart_Boundary_x{$semi_rand}x"; 
 
// headers for attachment 
$headers .= "\nMIME-Version: 1.0\n" . "Content-Type: multipart/mixed;\n" . " boundary=\"{$mime_boundary}\""; 
 
// multipart boundary 
$message = "This is a multi-part message in MIME format.\n\n" . "--{$mime_boundary}\n" . "Content-Type: text/plain; charset=\"iso-8859-1\"\n" . "Content-Transfer-Encoding: 7bit\n\n" . $message . "\n\n"; 
$message .= "--{$mime_boundary}\n";
 
// preparing attachments
for($x=0;$x<count($files);$x++){
	$file = fopen($files[$x],"rb");
	$data = fread($file,filesize($files[$x]));
	fclose($file);
	$data = chunk_split(base64_encode($data));
	$message .= "Content-Type: {\"application/octet-stream\"};\n" . " name=\"$files[$x]\"\n" . 
	"Content-Disposition: attachment;\n" . " filename=\"$files[$x]\"\n" . 
	"Content-Transfer-Encoding: base64\n\n" . $data . "\n\n";
	$message .= "--{$mime_boundary}\n";
}
 
// send
 
$ok = @mail($to, $subject, $message, $headers); 
if ($ok) { 
	echo "<p>mail sent to $to!</p>"; 
} else { 
	echo "<p>mail could not be sent!</p>"; 
} 
 
?>

If there is any interest I may provide a service where you provide me with your kindle email address and I’ll add you to the list to receive the unofficial version as well :).

Sources: PHP email with attachments