Wednesday, 23 December 2009

C++ IS useful - Click Once workaround

cpp[Concrete/ Quite Interesting] Me and C++, we’re old mates – the sort you find on Friends Reunited. I was fortunate to work with Microsoft and Glockenspiel in the late 80’s, working in MFC 1.0 and beyond. Later I would teach C++ for QA Training and others. I really loved the language and managed to build some sizeable solutions in C++, MFC and ATL.
Then came .Net and C# and for me as with many others, we could now start to focus on the problem domain, design and patterns to solution rather than pointers, memory and language syntax.
In C# almost every line of code looks cleaner and addresses the issue at hand, something C++ never seemed to achieve. Look at this simple example taken randomly from the MSDN:
HTTPWebRequest in C++
#using <System.dll>

using namespace System;
using namespace System::Net;
using namespace System::Text;
using namespace System::IO;

// Specify the URL to receive the request.
int main()
   array<string^>^args = Environment::GetCommandLineArgs();
   HttpWebRequest^ request = dynamic_cast<httpwebrequest^>(WebRequest::Create( args[ 1 ] ));

   // Set some reasonable limits on resources used by this request
   request->MaximumAutomaticRedirections = 4;
   request->MaximumResponseHeadersLength = 4;

   // Set credentials to use for this request.
   request->Credentials = CredentialCache::DefaultCredentials;
   HttpWebResponse^ response = dynamic_cast<httpwebresponse^>(request->GetResponse());
   Console::WriteLine( "Content length is {0}", response->ContentLength );
   Console::WriteLine( "Content type is {0}", response->ContentType );

   // Get the stream associated with the response.
   Stream^ receiveStream = response->GetResponseStream();

   // Pipes the stream to a higher level stream reader with the required encoding format. 
   StreamReader^ readStream = gcnew StreamReader( receiveStream,Encoding::UTF8 );
   Console::WriteLine( "Response stream received." );
   Console::WriteLine( readStream->ReadToEnd() );
HTTPWebRequest in C#
using System;
using System.Net;
using System.Text;
using System.IO;

public class Test
    // Specify the URL to receive the request.
    public static void Main (string[] args)
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create (args[0]);

        // Set some reasonable limits on resources used by this request
        request.MaximumAutomaticRedirections = 4;
        request.MaximumResponseHeadersLength = 4;
        // Set credentials to use for this request.
        request.Credentials = CredentialCache.DefaultCredentials;
        HttpWebResponse response = (HttpWebResponse)request.GetResponse ();

        Console.WriteLine ("Content length is {0}", response.ContentLength);
        Console.WriteLine ("Content type is {0}", response.ContentType);

        // Get the stream associated with the response.
        Stream receiveStream = response.GetResponseStream ();

        // Pipes the stream to a higher level stream reader with the required encoding format. 
        StreamReader readStream = new StreamReader (receiveStream, Encoding.UTF8);

        Console.WriteLine ("Response stream received.");
        Console.WriteLine (readStream.ReadToEnd ());
        response.Close ();
        readStream.Close ();
Ok, not a huge difference, but enough of a difference to make C# easier to write, easier to read and so we can focus better on the implementation. I really liked C++ but for me there is no reason to work so hard – C# makes coding more enjoyable and productive.
No reason until…

Click Once Deployment – sounds simple – NOT

I have been developing an Office Outlook Plug-in for Far2MuchMail is a solution to help people better manage their emails in Outlook and it’s free. What I wanted was to make it easy for people to download and install the plug-in from the Internet so naturally I thought ‘Click Once Deployment – just the puppy’. WRONG.
I want to encourage people to use my plug-in. Did I say it was free. Being free, the last thing I wanted was loads of messages during installation telling the user that the software was from an unknown publisher, couldn’t be trusted, would probably cause a fire in the house, sell your wife and kids to criminals, cause global warming, and so on. I mean, you would never install software, free or not, with that level of scare mongering.
I understood why the warnings were there and knew that if I bought a certificate, my problems would be over. But somehow I felt like I was being conned. A certificate felt like one of those Microsoft Taxes – like MSDOS. There had to be another way.
I knew I could create a certificate using MAKECERT. but to stop Click Once raising alarming messages, this certificate needed to be installed prior to installing the app.
I came across an article that talked about programmatically adding your certificate into the Trusted Installer list on a local machine. Great, all I needed to do was implement this and have it called at the start of installation as a pre-action.
But Click Once is locked down so I had to look at an alternative installer. I decided upon a standard Deployment Project in Visual Studio. Finally I was able to install my certificate prior to installing the plug-in. But there was a problem…

VS2008 and the Deployment Project Bootstrap.

Unfortunately, in VS2008 Microsoft changed the bootstrap program (setup.exe) as part of their deployment package. The change was subtle but important. Previous versions of the program would remain in memory until the installation ended. This allowed you to use a program like WinZip or IExpress to package the setup and MSI into a single executable. Setup.exe would remain in memory until the setup had completed so you could reliably let WinZip or IExpress wait until the setup had finished then automatically clean up any temporary files. Everything worked. In VS2008, the bootstrap setup.exe was designed to leave memory as soon as it had kicked off the MSI package. The result was the single package created by WinZip or IExpress would think the setup has finished and would start to pull temporary files, causing the setup to fail. Very frustrating.
I was nearly there, all I now needed to do was keep the package from terminating while the MSI was installing. C++ to the rescue.

WaitForProcess - C++ application

image I decided to use IExpress to package my setup into a single executable deployable from my website. IExpress allowed me to name the components of the installation including the setup.exe bootstrap and MSI package. It also allows you to define post installation steps. It is here I named a program called WaitForProcess, written in C++.
WaitForProcess is very simple, it continues to run if a named process is still in memory. So calling WaitForProcess and naming msiexec.exe in the installation package is all I need to do to ensure the IExpress package remain in memory until the MSI package has completed installation.
The trick was to write WaitForProcess without any dependencies on .NET which may not be installed until after the setup has completed. Indeed, I needed to rely on as few components as possible. C++ without ATL/MFC only requires standard libraries like USER32 and KERNEL.

So here is the code to WaitForProcess:

// WaitForProcess.cpp : Defines the entry point for the application.

#include "stdafx.h"
#include "WaitForProcess.h"
#include <psapi.h>
#include <shlwapi.h>

bool IsRunning(LPCTSTR pProcessName) //findProcess should be 1,2 or 3 and tells us which of the predefined processes to look for (see searchString array below)
    DWORD Processes[1024]; //Buffer array where enumerated processes will go
    TCHAR ProcessPath[MAX_PATH] = _T(""); //Path of the examined process
    HANDLE hProcess = 0; //Handle of the examined process
    DWORD bytesNeeded; //bytes written into buffer array by EnumProcesses
    int nProcesses; //Number of actually enumerated processes calculated from bytesNeeded
    int i=0;
    bool found=false; //Stop examining further processes if found is true
    if(EnumProcesses(Processes, sizeof(Processes), &bytesNeeded))
        nProcesses = bytesNeeded / sizeof(DWORD);
        while((i<nProcesses) && (!found))
                hProcess = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, Processes[i]);
                    GetProcessImageFileName(hProcess, ProcessPath, sizeof(ProcessPath)/sizeof(TCHAR));
                    found=!_tcsicmp(ProcessPath, pProcessName);
    return found;

int APIENTRY _tWinMain(HINSTANCE hInstance,
                     HINSTANCE hPrevInstance,
                     LPTSTR    lpCmdLine,
                     int       nCmdShow)

    //TCHAR sMessage[1024] = _T("");
    //wsprintf(sMessage, _T("Process: %s"), lpCmdLine);
    //::MessageBox(0, sMessage, _T("Wait For Process"), MB_OK);

    if (IsRunning(lpCmdLine))
        //::MessageBox(0, _T("Found"), _T("Wait For Process"), MB_OK);

    //::MessageBox(0, _T("Done"), _T("Wait For Process"), MB_OK);
    return 0;


Well, I still find C# a more productive language but it goes to show, when the going gets tough, the tough turn to C++.

Tuesday, 15 December 2009

Cosy Orbs – Web App Games in HTML5 for the iPhone

image [Concrete/Interesting] In this article I’m going to talk about HTML5 Canvas and graphics programming for web apps. I’ll be focusing on my experience and as way of example, I will be talking about ‘Cosy Orbs’ – a very simple, interactive graphics web application.

Installing Cosy Orbs

You can install Cosy Orbs onto your iPhone by following the link below in Safari:

Once the page has loaded, ‘Add to Home Screen’ by selecting the ‘+’ icon on the navigation bar at the bottom of the Safari page. Cosy Orbs will now be placed on your home screen and can be run by clicking the application icon. 

About Cosy Orbs

In 1999, the moon was knocked out of its orbit by a nuclear waste explosion, sending all 311 inhabitants of Moonbase Alpha into outer space. Also, like a lot of other people, I was looking at Java and Java Applets. Working my way through the SDK and samples, I came across a sample called ‘Graph Layout’. I was quite impressed with Java but I was blown over by Graph Layout. It was so cool. Basically, a string parameter to the applet defined a number of nodes on a network graph. So the string ‘Nigel-Aiesha, Aiesha-Sophia, Sophia-Nigel, Sophia-Melina, Melina-Aiesha’ would define a graph with four nodes (Nigel, Aiesha, Sophia and Melina). In addition, it defines the relationship between these nodes.

The applet would graphically represent these nodes as boxes connected by lines. The lines represent the relationship between the nodes. So, with the example above, the graph would look like:


But that wasn’t the cool bit.

The cool bit was the applet applies an algorithm to the graph that worked to minimize the stress between the nodes. The stress is defined as the length of the connecting lines, measured against a desired value. So if two nodes have a desired distance of 100 and the actual distance is 150, the stress is 50.

Every cycle, the stresses are calculated and the nodes are moved relative to the other nodes so as to reduce the stress.The result is the graph gracefully reorganizes itself into the lowest stress state by moving the nodes about each other.

Because of the nature of the algorithm and the fact that it acts across the entire system of nodes, the process of reducing the stress or ‘relaxing the graph’ is indeed very graceful.

This applet has sat in a dusty corner of my mind waiting to be re-applied or re-engineered. Then came iPhone, HTML5 and Canvas and Cosy Orbs was born.

Cosy Orbs uses a similar algorithm to relax the stress across a network of nodes, but:

  • It is written in HTML5, JavaScript, jQuery
  • It is an iPhone Offline web app.
  • It is more graphical with ‘Orbs’ or shiny spheres rather than boxes with text
  • It uses a Touch enabled UI rather than mouse.

So how’s it all done?

Offline Web App

This is a must have feature for any web app, especially a game or fun app. No problem on the iPhone, with HTML5 all you need to do is set up a manifest file. Well, not that simple but I’ve blogged about it before so won’t repeat myself here. See iPhone Offline Web Apps – the RESTful Way – Part 1 of 2.

Initializing the Canvas

The Canvas is an HTML5 element that represents a drawing surface. In essence it is a bitmap. HTML5 currently support ‘2D’ drawing operations and some other useful functions over the canvas like streaming the canvas to a string.

I use the following code to initialize the canvas:

myCanvas = document.createElement('canvas'); = 'myCanvas';
myCanvas.width = 300;
myCanvas.height = 300;

After creating the canvas, the element is appended to a DIV on my page. I also remove a compatibility statement. This statement is shown if you run the app in an incompatible browser, otherwise it is removed:

<div id="notice" style="text-align:center"><strong>This application is designed for the iPhone. Link to this page in Safari on your iPhone.</strong></div>

Preloading the Orbs

Unlike the Java Applet, I am representing nodes with lovely coloured spheres. I have five different images for my Orbs; blue, red, green, purple and yellow. So when drawing a node, I will assign it one of the colours.

When drawing I want to work with Image elements, so I need to preload the images before any drawing starts. This was a bit tricky because loading an image requires going to the server to fetch the image data, so the Image.onload() is asynchronous. Because I wanted to wait for the images to finish loading before starting any drawing, I decided to set up a simple timer:

var imageString = 'images/orb-blue.png,images/orb-green.png,images/orb-purple.png,images/orb-red.png,images/orb-yellow.png';
var imagesToLoad = 0;
function initImages()
    function(idx, val) {
      var img = new Image();
      img.onload = function() {imagesToLoad--};
      img.src = val;
function completeImageLoad()
  if (imagesToLoad > 0)
    // Ready to go

The images are all named in the string ‘imageString’. When loading, I simply split this string, and for each image named, I create an Image element and load from the source. I also increment a load count, counting each image as it’s loaded. Finally I add the image to the list of preloaded images - this will be used when drawing the images.

During the Image.onLoad(), which is fired asynchronously once the image has loaded, I decrement the load count, signalling that this images has loaded completely.

Once all the images have been processed (but not necessarily completed loading), I call a complete method. This is called asynchronously using the setTimeout() method.

Now all I need to do is poll for the load count to reach zero, which will happen when all the images have completed loading. If the images are still loading, the complete method is called again under timer.

Once all the images have been loaded, I need to set up the nodes…

Initialize the Nodes and off we go

Nodes and their relationships are defined by a string called ‘edgeString’. The process of initializing the nodes involves reading this string, and building a list of Node() and Edge() objects. Each Node() object holds the canvas co-ordinates of the node and the image assigned to the node – these are used when drawing the network. Each Edge() object defines the stress between two Nodes(). As a node is created, an image from the list of images is assigned to the node in a round robin fashion.

Once the Nodes() and Edges() are initialized, the network is rendered to the canvas.

Finally, a timer is set to refresh the display:

function drawCanvas()
  setTimeout('drawCanvas()', refresh);
  if (myCanvas.getContext) {
    var ctx = myCanvas.getContext('2d');
    ctx.fillStyle = 'rgb(0,0,0)';
    ctx.lineWidth = 2;
    for(var j=0; j<nodes.length; j++)
      ctx.drawImage(nodes[j].img, nodes[j].x-21, nodes[j].y-21);

When fired, a function is called to Relax() the graph – that is, to process each node’s stresses,  incrementally reducing the stress across the network. Once the stress has been reduced and the network is a little more relaxed, a call is made to render the network to the canvas. This happens every 150ms, which provides a smooth animation with the network slowly relaxing into its lowest stress state.

Rendering is simple, first, a ‘2d’ context is obtained on the canvas. With this context, the canvas is cleared with a black background through a call to the fillRect() function. Then for each node in our array of Nodes(),  the image on the node is drawn to the canvas using the drawImage() method.

What about Touch Events

Not much fun if you can’t interact with the Orbs, so I implemented a simple touch interface:

$('#myCanvas').bind('touchstart touchmove touchend', function(evt) {
  var touch = evt.originalEvent.targetTouches[0];
  switch (evt.type) {
    case 'touchstart':dragNode = findNodeByPos(touch.pageX-this.offsetLeft, touch.pageY-this.offsetTop);break;
    case 'touchend':dragNode = null;break;
    case 'touchmove':
      if (dragNode) {
        dragNode.x = touch.pageX-this.offsetLeft;
        dragNode.y = touch.pageY-this.offsetTop;

To do this I have to bind the touch events, touchstart, touchmove and touchend, to a callback function for the named element. jQuery make life very simple through the use of the bind() method. 

In the event handler, the event type is determined and for touchstart events, the nodes are searched for a match to the touch position. That is to say, the nodes are searched for any node under the user’s finger.

Touchmove tracks the movement of a finger on the iPhone surface. I use this to update the screen position of the selected node (if one was found during touchstart). Finally,  touchend stops the dragging operation. Job done.


CosyOrbs, not exactly a game for the iPhone, but I am impressed as to how easy it is to create interactive graphics under HTML5.

Tuesday, 1 December 2009

iPhone Offline Web Apps – the RESTful Way – Part 2 of 2

html5REST [Concrete/Very Interesting] Ok, so hopefully you got the static application working offline. I was quite satisfied when I got it working, though not nearly as much fun as watching Push in HD on my Samsung 6000 LED TV (thinner than my laptop – wow).

Now for the RESTful part. We need to make REST calls when online and work from a cache when offline.






Cache Manifest White list

As well as defining the resources that are cached, you can specify resources that should not be cached in your manifest file:

# Version

Any resources named under the ‘NETWORK:’ label are considered white listed and should not be fetched from the cache.

Notice that you can name cached resources after a ‘NETWORK:’ section by using the ‘CACHE':’ label.

Naming white listed resources allows us to access resources in our web app that are not in the cache. This solves a serious problem for us. Imagine trying to add your first service call to an offline application with static only resources. You can’t name the service in the manifest as a cacheable resource. This wouldn’t work the first time you went offline and the app would fail to load with a ‘Internet not available’ error. If you omit the resource, the manifest is wrong and you will end up running the cached app, the one without the service call. Real bummer.

‘NETWORK:’ specifies a non-cached resource and one without fallback. There is a ‘FALLBACK:’ label that allows us to specify a non-cached resource and cached alternative. Useful if you want to put up a big banner when offline saying ‘Sorry, come back later’ but not what we are trying to do here.

One final point about the white list before I move on. Unlike the named cached resources, the white list is in fact a resource prefix. So, in the example above, I named my REST service, not the individual methods. This is useful because our methods are likely to have parameters and we don’t know necessarily know what the endpoint looks like until runtime.

The HTML5 Database

So now to storing the results of our RESTful calls. The solution lies in another HTML5 feature – HTML database. In Safari and on the iPhone this means SQLite.

I’m a real fan of SQLite. Great small footprint relational database. I’ve been using it for some time now under .NET as storage for my Outlook plug in.

Storing Service Results

I chose a very simple schema for my database – simple Key/Values in a table using TEXT field types. SQLite, jQuery and HTML5 make it very easy for me:

function initDatabase()
  db = openDatabase('iRetailPassport', '1.0', 'iRetailPassport', 1000000);

A couple of important points about the code above:

  • CREATE TABLE IF NOT EXISTS is very useful and part of the zero administration nature of SQLite. This statement means I can create and open a database in a single statement. And the transaction succeeds even if the database already exists.
  • CONFLICT REPLACE is also very useful. It allows me to do an insert even if the record exists. Also knows as an UPSERT or MERGE operation, this save time and code later on.

Because I’m making JSON service calls, the call results are both efficient and ready to store in my database:

  type: "GET", url: "/iretailpassportsvc/clientservices.svc/passport",
  dataType: "json",
  cache: false,
  timeout: 1000,
  success: function(passport)
        transaction.executeSql("INSERT INTO settings (k,v) VALUES ('passport',?);", [passport], null);
  error: function()

Notice the jQuery AJAX call to the server is marked as JSON. It is also important to set a timeout to ensure this function returns if the server is unavailable.

The AJAX method has both ‘success:’ and ‘error:’ callbacks. In the error callback, I’m handling the error. A likely implementation here is to start some sort of dialogue with the user when the event is fired and to cancel that dialogue on error. A dialogue could be information on a status or progress bar for example.

The success callback does the work of posting the JSON response to the database. Because I used ‘CONFLICT REPLACE’ in the database definition, the ‘INSERT’ will be a merge insert.

The success case would also update the user interface with fresh data but I have omitted this from the example to keep things simple.

Storing Images

In my application I was making a call to the REST service to get an image of a barcode that represents the users’ unique identifier. My original design was based on making this call as and when required and for the service to generate the image at run time and stream the image to the client. The strategy to generate at run time meant I could keep storage requirements low at the server and meant I didn’t have to worry about any referential integrity between the user identifier and the image. I wanted to keep this strategy for the offline application and reuse the service interface.

This meant storing an image streamed by the service. Well again, HTML5 comes to the rescue with the <canvas> element.

A canvas is a bit oriented drawing area – a bitmap.

The DOM supports a number of drawing operations to the canvas and also provides methods for serializing the canvas:

var src = "/iretailpassportsvc/clientservices.svc/barcode/" + passport;
var canvasImg = document.createElement("canvas");
var serializedVal = 'data:,';
var img = new Image();
img.onload = function() 
  canvasImg.width = img.width;
  canvasImg.height = img.height;
  canvasImg.getContext("2d").drawImage(img, 0, 0);
  serializedVal = canvasImg.toDataURL();
      transaction.executeSql("INSERT INTO settings (k,v) VALUES ('barcode',?);", [serializedVal], function (transaction, results){setTimeout("jQT.goBack()", 1000)});
img.src = src;

This is very cool. In the code above I am creating <canvas> and <img> elements. I’m then loading the image using a reference to the image service endpoint with img.src = src.

During the image onload() event I set up the canvas dimensions to match that of the image (remember, the canvas is really a bitmap) and then draw the image to the canvas with the drawImage() statement.

The result of this is the canvas is now a bitmap representation of the image loaded from the server.

Finally, I serialize the image to a data URL. A data URL is a BASE64 representation of a resource and can be used anywhere a resource is used. So in this case it will be something like:


So now I have a serialized string representation of the image generated at run time by my service. All I need to do now is save it to the database.

Later on, I can load this string  and use a simple statement like:

img.src = data

to display the image.

Asynchronous Everywhere

Remember that calls to the database and AJAX are asynchronous. Methods around these calls provide callbacks and you need to put your code inside the callback or you will see some very odd behaviour.

I got caught out when handling the image serialization. The call to load the image is delayed because the source is a network resource. So the img.onload() may take time to complete. My mistake was to write the serialized canvas data to the database in code following my img.src = URL. This had very strange results and was saving incomplete data (thank goodness for Web Inspector or I may still be sitting scratching my head).

The fix was to put all the serialization to database inside the img.onload().

Offline/Online Data Strategy

Now we know how to white list service calls, where to store the call results and how to store these results. One final point is around the offline/online data strategy.

A key benefit of a RESTful architecture is to provide fluid network applications without the need for holding vast datasets client side. So, if you need data or need to transition state, you call a service method. An offline model goes against this and necessitates storing data working with aged data.

But offline working cannot be ignored on mobile devices – just try using the internet on the train or underground.

The trick is to strike a balance between offline data fetches and online service fetches.

One strategy for offline/online data is to always make a service call and if the call fails or there is a timeout, read from the database. This helps keep the age of the data low but will result in a choppy user interface with a lot of ‘Please Wait’ timers.

A better strategy is to do things the other way round. That is, always read from the database. At the same time, issue a service call. If the call comes back with data, update the cache and user interface. To help with usability, you can always display a ‘Refreshing…’ message to the user is a status area.


As you can see from the this blog, the theory of offline RESTful applications is not a big challenge. The implementation raises a number of pitfalls which I hope I have been able to highlight.

But when it works, believe me…It’s all worth it.