First up, we’re doing a talk on R, Shiny and Web Applications @ Chippenham Tech Chat on Weds June 19th 2013. More info here:

If you’ve not come across it Shiny is a reactive programming web application framework built around R for exposing analytic applications.

It’s made by the guys from RStudio, who also provide the great IDE that lots of us use.

An introduction to Shiny from RStudio:
http://rstudio.github.io/shiny/tutorial/

An introduction to Shiny from the Dallas R users group here:
https://github.com/trestletech/shiny-sandbox/blob/master/shiny_introduction.pdf

and a good overview of deploying Shiny on Amazon EC2 here:
http://www.r-bloggers.com/deploying-shiny-server-on-amazon-ec2/

I guess there’s a few notable things from my side that I thought were pretty great when working with Shiny.

Installation

Get up and running by installing with the following:

install.packages("shiny", repos="http://shiny.rstudio.com")

and then run the example (starts a webserver on port 8100 and invokes xdg-open on Linux)

library(shiny)
runExample("01_hello")

Application Structure

The application consists of two main parts:

User Interface – ui.R
* Layouts
* Panels
* Inputs (selectInput, numericInput etc…)

Serverside – server.R
* function listeners

Architecture

The Shiny architecture is made up of a number of parts internally, namely a set of web/browser components including:

• Twitter Bootstrap – a UI library that provides scaffolding in terms of layouts with HTML and CSS
• Websockets – a way to do bi-directional communication with a browser/webserver.

Serverside Shiny makes use of Node.js for hosting (it doesn’t have to), but this to probably the most scalable approach.

Finally, it’s based on the concept of reactive programming which provides a number of parts:

• Triggers/Inputs – Source (Shiny Reactive Value)
• Sinks/Outputs – Endpoint (Shiny Observer)

The triggers are executed when dependencies (inputs) change resulting in a change to the output:

shinyServer(function(input, output) {
 output$plotOut <- renderPlot({
   hist(faithful$eruptions, breaks = as.numeric(input$nBreaks))
   if (input$individualObs)
     rug(faithful$eruptions)
 })
 
 output$tableOut <- renderTable({
   if (input$individualObs)
     faithful
   else
     NULL
 })
})

In some cases, changes could take a long time to run (accessing database, read from file). Shiny proposes conductors to encapsulate these operations. Shiny calls these reactive expressions

More info here – http://rstudio.github.io/shiny/tutorial/#reactivity-overview

Deployment

In terms of deployment you have a number of options:

1. You can deploy apps on hosted Shiny Server

2. Run from a Github paste (Gist) or from Github (this is an awesome feature)

shiny::runGist('3239667')
shiny::runGitHub('shiny_example', 'rstudio')

3. Deploy on Amazon EC2

4. Host yourself by installing Apache and NodeJS

Hope that gives you a good overview of Shiny, come along to the meetup if you want to learn more about it.

Recruitment Plug Here ->

Yeah, yeah we’re also recruiting for Java developers, Javascript developers for our devops and development teams here:

http://www.mango-solutions.com/careers.html

plus R developers : )

Our Stackoverflow company page is here – http://careers.stackoverflow.com/company/mango-solutions

http://www.playframework.com/documentation/2.1.0/ScalaTodoList

It’s not exactly the most straightforward written tutorial I’ve come across from a framework so I thought I’d post the full sources here along with an archive of everything:

Download

for people to refer to if they get stuck.

One other thing, it doesn’t tell you to enable database evolutions by default in the application.conf file so make sure you do, it’s not a showstopper by any means, but it’s handy to know.

The application consists of the following sources:

* app/controllers/Application.scala – Main Controller
* app/models/Task.scala – Task Companion Object
* app/views/index.scala.html – Index Template
* conf/application.conf – Configuration
* conf/routes – Routes
* conf/evolutions/default/1.sql – Database Evolution/Migration

Application.scala

package controllers
 
import play.api._
import play.api.mvc._
import play.api.data._
import play.api.data.Forms._
import models.Task
 
// Controller
object Application extends Controller {
 
	val taskForm = Form(
		"label" -> nonEmptyText
	)
 
	def index = Action { 
		Redirect(routes.Application.tasks) 
	}
 
	def newTask = Action { implicit request =>
	  taskForm.bindFromRequest.fold(
	    errors => BadRequest(views.html.index(Task.all(), errors)),
	    label => {
	      Task.create(label)
	      Redirect(routes.Application.tasks)
	    }
	  )
	}
 
	def tasks = Action {
	  Ok(views.html.index(Task.all(), taskForm))
    }
 
	def deleteTask(id: Long) = Action { 
		Task.delete(id)
  		Redirect(routes.Application.tasks)
	} 
}

Task.scala

package models
 
import anorm._
import anorm.SqlParser._
import play.api.db._
import play.api.Play.current
 
case class Task(id: Long, label: String)
 
// Task companion object
object Task {
 
	val task = {
	  get[Long]("id") ~ 
	  get[String]("label") map {
	    case id~label => Task(id, label)
	  }
	}
 
	def all(): List[Task] = DB.withConnection { implicit c =>
  		SQL("select * from task").as(task *)
	}
 
	def create(label: String) {
	  DB.withConnection { implicit c =>
	    SQL("insert into task (label) values ({label})").on(
	      'label -> label
	    ).executeUpdate()
	  }
	}
 
	def delete(id: Long) {
	  DB.withConnection { implicit c =>
	    SQL("delete from task where id = {id}").on(
	      'id -> id
	    ).executeUpdate()
	  }
	}
}

index.scala.html

 
@(tasks: List[Task], taskForm: Form[String])
@import helper._
@main("Todo list") {
 
    <h1>@tasks.size task(s)</h1>
 
    <ul>
        @tasks.map { task =>
            <li>
                @task.label
 
                @form(routes.Application.deleteTask(task.id)) {
                    <input type="submit" value="Delete">
                }
            </li>
        }
    </ul>
 
    <h2>Add a new task</h2>
 
    @form(routes.Application.newTask) {
 
        @inputText(taskForm("label")) 
 
        <input type="submit" value="Create">
 
    }   
}

conf/routes

# Routes
# This file defines all application routes (Higher priority routes first)
# ~~~~
 
# Home page
GET     /                           controllers.Application.index
 
# Map static resources from the /public folder to the /assets URL path
GET     /assets/*file               controllers.Assets.at(path="/public", file)
 
# Tasks          
GET     /tasks                  controllers.Application.tasks
POST    /tasks                  controllers.Application.newTask
POST    /tasks/:id/delete       controllers.Application.deleteTask(id: Long)

application.conf

...
db.default.driver=org.h2.Driver
db.default.url="jdbc:h2:mem:play"
...

conf/evolutions/default/1.sql

# Tasks schema 
# --- !Ups
 
CREATE SEQUENCE task_id_seq;
CREATE TABLE task (
    id INTEGER NOT NULL DEFAULT NEXTVAL('task_id_seq'),
    label VARCHAR(255)
);
 
# --- !Downs
 
DROP TABLE task;
DROP SEQUENCE task_id_seq;

Hope that helps !

Nowadays, our modern browsers, Chrome, Firefox and Opera have HTML5 canvas implementations that provide a mechanism for doing 2D graphics that are beginning to approach the capabilities of what Adobe Flash provides – TweenJS (http://www.createjs.com/#!/TweenJS) provides advanced animation and support for tweening whilst demos such as the Madmanimation really show off what is capable of the browser with CSS and HTML5. If you haven’t seen it then check it out:

http://stuffandnonsense.co.uk/content/demo/madmanimation/

R Based Graphics

We (Mango Solutions) do lots of work with the open source language R for graphing and charting serving up static images within our applications for advanced reporting and analytics purposes. Providing interactive mechanisms for this is also possible.

http://www.rstudio.com/shiny/

Check our R Shiny from the RStudio guys and you’ll see how they’ve used HTML5 webworkers and pooled R sessions. Demos here:

http://glimmer.rstudio.com/winston/stocks/

Interactive Javascript Based Visualisation

There are also some very nice fully HTML5, CSS compliant interactive libraries. Top of the pile (in my opinion) by far is D3 (a successor to Protovis) that provides a way to create sophisticated interactive visualisations.

See the Vornoi diagram example here:

http://bl.ocks.org/1734663

and the combination of D3 with the Crossfilter Javascript library for large multi-variate datasets within the browser:

http://square.github.com/crossfilter/

Realtime Graphing and Charting

Realtime graphing (particularly for time series data etc…) is fairly recent. Our first look at this was with a cool little library called Flot.js:

http://www.flotcharts.org/

Another option is HighCharts which provides a wide set of options for graphs and charts in Javascript:

http://www.highcharts.com/

This works nicely for basic charts as well and the best bit is that you can use it from Internet Explorer 6 and upwards, performance is obviously an issue in terms of browser and version.

The dynamic charts appear to make use of SVG extensively for display, example here:

http://www.highcharts.com/demo/dynamic-update

Big Data In Bristol

There’s a big data meetup in Bristol this Friday and Saturday (18th January 2012) to do some hacking on BigData using things like Hadoop (HDFS, MapReduce), Shark and Spark. Finishing this off with a visualisation with D3 would be a nice touch. More info on the meetup here:

http://www.meetup.com/south-west-big-data/events/95899692/

Also, we’re hiring for a Javascript developer currently at Mango Solutions, check out the job posting here:

http://www.jobserve.com/gb/en/search-jobs-in-Chippenham,-Wiltshire,-United-Kingdom/JAVASCRIPT-DEVELOPER-8A8BC205FB4E4766/

There’s also an opening for Senior Java Developers working with xText, DSL’s, enterprise messaging and integration.

cd /usr/local/subversion
cp pre-commit.tmpl pre-commit
chown apache:apache pre-commit
chmod +x pre-commit

Pre-commit hooks generally call out to other programs to do their work. It’s a Bash script by default, but people tend to write scripts in Perl/Python etc… as it’s a bit more readable than plain old Bash.

You can use pre-commit hooks for a number of reasons:

• To enforce that users enter comments on commit messages
• To enforce that users specify issue ID’s on commit messages to match tickets in bug trackers such as Bugzilla, Jira or Mantis
• Running style checking
• Preventing BOM’s in source code
• Sending email notifications to a group of people

There’s a fair few good articles already on this:

• Overview of the whole process – http://www.agileweboperations.com/leveraging-subversion-hooks-for-fun-and-profit
• Some example scripts written in Bash – http://blog.grimsy.net/2008/07/a-few-svn-pre-commit-hooks/
• Perl based pre-commit script – http://shah-oss.blogspot.co.uk/2011/04/pre-commit-hook-script-for-subversion.html

One final note, under Windows things are a bit different, only in the sense that you have to use a batch file instead of a bash script. I ran into this with VisualSVN and ended up hacking the following Python script and Batch file together to solve the problem:

import sys;
import os;
import re;
 
svnlookcmd = sys.argv[1];
txn = sys.argv[2];
repo = sys.argv[3];
logcmd = "svnlook log -t " + txn + " " + repo;
 
line = os.popen(str(logcmd)).read()
 
if re.match(r'\b(?:bug|issue|task)\s*[#]{0,1}(\d+)\b', line, re.I):
    print "Found valid log message: " + line;
    exit(0);
else:
    print "Error checking log msg: " + line;
    exit(1);

setlocal
 
set SVNLOOK="%VISUALSVN_SERVER%\bin\svnlook.exe"
set REPOS=%1
set TXN=%2
 
echo "Executing precommit check" >&2
python c:\repos\scripts\checkcommit.py %SVNLOOK% %TXN% %REPOS%
set RESULT=%ERRORLEVEL%
echo "Executed check with result: " + %RESULT% >&2
IF %RESULT% == 0 GOTO :NORMAL_EXIT
REM Block commit by default
echo "Commit msg must reference an issue number in the format: issue 1234 OR bug 1234 OR task 1234" >&2
exit /b 1
REM All checks passed, so allow the commit.
 
:NORMAL_EXIT
exit 0
 

To do it I used a simple PHP5 script with the curl dependency installed with:

sudo apt-get install php5-curl

along with a bit of XPath, which can be customised whatever way, script is as follows:

<?php
$curl = curl_init('http://www.kickstarter.com/projects/1461411552/elite-dangerous');
 
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0');
$html = curl_exec($curl);
curl_close($curl);
 
$dom = new DOMDocument();
$dom->loadHTML($html);
$xpath = new DOMXPath($dom);
$results = $xpath->query("//data[@data-currency='GBP']");
 
$pre = "<h1>";
$post = "</h1>";
 
foreach ($results as $r){
	$amountstr = $r->childNodes->item(0)->nodeValue;
	Print $pre . "Total Funding: " . substr($amountstr, 1) . $post;
}
?>

Customise as you see fit, it serves as the beginning for a simple update widget in Ubuntu or other mechanism.

http://www.meetup.com/Chippenham-Tech-Chat/

It takes place at the office of Mango Solutions, the company I work for here:

View Larger Map

The first meetup is on the 18th September 2012, come along if you want to find out more, topic still to be decided.

The following script will remove all the workspace builds from Jenkins that exist, reset the build number for all projects specified
in the .hudson/jobs directory.

Use it with care. I take no responsibility for misuse of this or bugs etc…

On another note, there is a plugin available for Jenkins that allows you to change the build number, see: http://wiki.hudson-ci.org/display/HUDSON/Next+Build+Number+Plugin

#
# Hudson build cleaner
# Cleans out all builds for projects that exist in the directory
# Resets the build number to 1
# Backup all projects before running this
 
function cleanbuild {
 
  echo "cleaning $1"
  cd $1
  ls -l
  rm -Rf builds/*
  rm -Rf lastSuccesful
  rm -Rf lastStable/
  rm -Rf workspace/*
  rm -Rf modules/*
  rm -f lastSuccessful lastStable
  rm -f nextBuildNumber
  touch nextBuildNumber
  echo 1 >> nextBuildNumber
  cd -
}
 
for project in `find -L -maxdepth 1 -type d -name '*' ! -name '.*' -printf '%f\n' | column`
do
  cleanbuild $project
done

However, it didn’t prove to be the most straightforward of installs. Just a few tips here based on what I did:

I checked out the source code from the trunk of the Subversion repository. The reason I did this was because the binary that was installed didn’t include a PMS.conf, not sure whether I could have manually created it.

svn checkout http://ps3mediaserver.googlecode.com/svn/trunk/ ps3mediaserver-read-only

then proceeded to build it with ant form the Apache website.

ant

After that it was a case of configuring it. It appears that it installs the PMS.conf file in:

$/.config/PMS/PMS.conf

so you need to go there to configure it.

I had to change the network interface that it bound on and the port used.

That proved to be a pain as none of the configuration options are documented. I ended up digging out the PmsConfiguration.java file from the source, link here:

http://code.google.com/p/ps3mediaserver/source/browse/trunk/ps3mediaserver/net/pms/configuration/PmsConfiguration.java

That contains all the names of the keys you need so for my purposes I needed the following in PMS.conf:

network_interface = eth0

Starting this up with:

pms.sh &

and scanning via the Playstation 3 Video -> Detect MediaServers brings up my PS3MediaServer.

There’s also a help forum here:

http://www.ps3mediaserver.org/forum

If you’re using phpmyadmin to update the database directly the only two changes you have to make are to the site_url and home option-name‘s in wp_options. Make sure you change the option_value for each of these to the new address.

One of the problems you may come across is that your old posts may reference the old domain in same way. If you want to update these then it’s laborious and manual task of editing them in the database – which can be error prone, unless done carefully. You can run a SQL update to update the domain in the post_content column of the wp_posts table as follows:

UPDATE wp_posts SET post_content = REPLACE(post_content, 'olddomain.com', 'newdomain.com');

I used this method to update the guid column to reference my new domain instead of my old domain.

Make sure you backup your database before running this. It’s a pretty easy process to do the backup of a MySQL database.

mysqldump -u username -p -v old.db > old.sql

If you’re also renaming your database in relation to your domain name move you can use the following:

mysqldump -u username -p -v old.db > old.sql
mysqladmin -u username -p create newdb
mysql -u username -p newdb < old.sql

http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/

Installation involved downloading the hadoop distribution and creating the right configurations (core-site.xml, hdfs-site.xml and mapred-site.xml). It also involved formatting the namenode and starting everything up. It’s a pain to do manually so I wrote a script to do the whole lot to save time. A few caveats:

• You need to set the JAVA_HOME location on your machine at the top of the script
• Specify the distro name (hadoop-0.20.2) to match the apache download link
• Configure passwordless SSH for your local machine

You can configure passwordless SSH as follows:

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa_for_hadoop
cat ~/.ssh/id_dsa_for_hadoop.pub >> ~/.ssh/authorized_keys

Hopefully that’ll save you a bunch of time! : )