Handling survey data : XML or JSON?

For a few weeks I have been pondering how best to handle 'survey data' for an organization that collects a lot of survey data and needs to make it available to people for a variety of 'business intelligence' needs.

The key needs are:

- must be in a format that is easy for the contributor of the survey data

- must allow for the same surveys to be repeated over time e.g. quarterly survey, for the same survey respondents e.g. Company ABC completes survey a number of times over a period of time

- must allow for survey results to be easily compared and contrasted e.g. same contributor, different survey submissions ( this quarter versus last quarter) and different contributors for the same time period e.g. Company ABC versus Company XYZ

- allow for the content of surveys to change over time e.g. questions can be added, deleted, changed

- allow people to access the data as surveys e.g. all data for a given survey's instance (all data for this quarter) or accessing the data for a given question over time e.g. answers to question 4 this quarter versus last quarter

- allow easy mapping of survey data to broader business concepts e.g question 2's answer is 'annual revenue' for a given respondent 

*whew* I think that's about it.... easy, huh?

much of the  research stuff I read suggested:

- XML is a good way of representing a survey and collecting survey data

- translate the XML data into a relational data model view of the survey and store in a SQL database

... sounds great but nothing a read really convinced me there could be a simple, generic approach that would work for all situations.

Keeping the data in an XML format, to allow data to be accessed in its survey form seems to hold some promise for addressing some of the requirements, at least partially, but thability to easily query the XML data seemed clumsy, at best... enter JSON

JSON seems to provide similar advantages for receiving survey data in a structured way and, with the advent of modern document databases, also the ability to easily query the data in its survey form

more to follow in this series... :)

Virgin Business Class to Brisbane

Today I am flying Virgin Business Class to Brisbane :)  I must say its a great experience so far Here's breaky ... Bircher muesli with macadamia nuts and yoghurt, fresh fruit, sparkling mineral water and a coffee to come *yum*

I'm in seat 1C, sitting in front of Dawn Fraser, trying to work out if I should say 'hello' :)

I'm catching up in Bris-Vegas and Gold Coast with some good friends I have not seen in years... looking forward to it :)

I gave up a chance to sit next to Dawn

Today I am flying to Brisbane, to catch up with some friends I have not seen in a while.  When I was checking in for the flight the lady at check in asked me if I preferred row 1 or 2, window or aisle.  I had pre booked row 1 on the aisle so I stayed with that. Once on the flight, seated and ready for departure, on walked Dawn Fraser and she quickly found her way to row 2.  There's a vacant seat next to her... that could have been mine *sigh*

Oh well, I'm sitting in front of Dawn instead,,,, maybe the title of this blog should be.... "Glenn beats dawn to finish line in Brisbane!" :D

Next best thing as being at Strata Hadoop World 2013 Conference in New York

I reviewed a video series from O'Reilly recently: Strata Conference New York + Hadoop World 2013: Complete Video Compilation - Tools and Techniques That Make Data Work

I have not yet watched all of the 71 hrs+ of conference content but from what I have seen I have been absolutely blown away by this series.  The quality of the content and format makes it the next best thing to being there.

On my Mac, the conference sessions download as QT movies.  The presentation from each speaker appears in a main window on the left and there is a smaller window in the top right where you can see the speaker.  The quality of both the video and audio is first rate and I was able to stream the content to my Apple TV and watch it on a big screen and it was like having front row seats at the conference.

One day I would like to get to a Strata Conference and this video series from O'Reilly has made me want to go even more.  However, I don't think I have missed any of the content last year by not being there.  For much, much less that the price of the airfare, I now have the whole conference program to watch.

PROS
Quality of content
Able to watch any sessions from the conference I choose

CONS
Presentations are large so make sure you have enough disk space and bandwidth

Bottom line:  Yes, I would recommend this to a friend!

Game theory can be fun

I reviewed a book recently:  Theory of Fun for Game Design, 2nd Edition by Raph Koster

I have always been an aspiring game designer and developer and thought this would be a good one to read to get a few ideas.

When I think of 'theory' I usually envisage something that, while useful, is going to have a hypotheses and some research and analysis... a stereotype I know, but I dot usually expect it to have any fun. Despite its title, this book bucks my stereotype.

The author's style is easy to read and 'conversational' and gives good practical examples of the 'theory', supported by lots and lots of nice hand drawn illustrations.

One of the points made in the book is that fun games are a form of 'pleasurable learning' and I certainly had fun reading this book and feel like I learned a lot.

As an aspirational game designer, I am not sure how I would apply what I have learned as there appears to be no 'formula' for designing fun games (and maybe that is just as well) but there are a lot of good stories and examples to get you thinking and this is what makes the book a good read.

PROS
Easy to understand
Helpful examples
Well-written

CONS
Might Be Hard To Apply The theory

Bottom line:  Yes, I would recommend this to a friend

Too sexy for my job?

I found this article on being a Data Scientist: The sexiest Job no one has

It could be a case of 'the grass is greener' syndrome but this is the job I want and I think Im on the way to getting it :) ... to me, its an exciting prospect that 'the universe (is) one large data set' ... *big data* indeed ;)

Now here come the 'damned lies and statistics'... Gartner estimates that there will be 4.4 million IT jobs created to support data analysis in just the next two years... and about half of those will be outside the US... I wonder what the stats say for data scientist jobs being generated in Australia?

I wonder what qualifications and experience I will *really*need, maybe its all that geeky stuff I keep hearing about...

- watch The Big Bang Theory religiously ... maths, science, history, unravelling the mystery ... (tick)
- can recite the script of The Matrix, 1, 2 and 3 (tick)
- think that leela and amy are hot (tick)
- yada, yada, yada (tick)

... oh, oh, what's this....

- PhD is advanced statistics

... darn, I was really close... well there's something to work on!

I just hope this 'wave' is really and not all sexy hype... cue LMFAO ;)

Technology leadership and innovation at Google - how does this relate to 'normal' organisations?

I came across the following article by Google on 'the 8 pillars of innovation' a few weeks ago and it got me thinking... what is innovation and technology leadership for a 'normal' company?

'Normal' companies are not Google... but there are many companies that would *love* to be 'like Google' (at least in some respects e.g. market cap, revenue streams, etc.) and so do some of the ideas presented in the article apply?

In many ways, companies like Google, and Apple, and others, are synonymous with innovation. The visible innovation occurs in the market, with new products and services that are offered. The less visible innovation is happening on the inside - as illustrated in the article and also in Hollywood films like 'The Internship' ;) lol

Its hard to imagine a 'disconnect' between the internal culture and the external culture. They may not be the same in every respect but there is no doubt IMHO that there is significant congruence between the two.

I think that is one reason why when I see a focus on client-facing-innovation I get excited because it probably means that there is a very innovative company on the 'inside'.  It makes me want to be part of it :)

One of the interesting points made in the article is how Google handles 'failure' - never fail to fail!  That part reminded me a lot of when I first became a consultant, at Simsion Bowles & Associates (good 'ol SBA)... Graeme (Simsion) did not mind failures, they were a chance to learn, the only rule was 'fail fast and learn quickly' ;) SBA was a very innovative company, at least in my own experience to that point, and its was a really fun place to work and to grow.

As an IT guy, one of the 'spin offs' I look for in an innovative company is how innovative they are in their use of technology.  I just hate it when IT is seen as 'pure expense' or a 'necessary evil' that makes it really hard to enjoy my work... in my career I have had a gut full of 'bureaucratic IT departments' pushing their view of how technology can and can't be used... the worst thing is its done from a purely 'selfish' or 'ignorant' 'don't create work for me' point of view :-/

As a disclaimer, let me say that last part is not a criticism, it is more of a challenge!

Every company needs to have some control over their technology, its cost, impact, etc. but I think it sometimes goes too far... people so far removed from the 'customer' think they are in charge and that they can make up the requirements... this is where it all goes pear shaped IMHO.

The 'effect' of client-facing-innovation is a catalyst to putting the focus squarely back on the client and their needs and when this happens it can help 'obliterate' the way companies think about technology (to borrow an expression from Michael Hammer in his famous 'don't automate, obliterate' article in the HBR).

The 'spin off' winners are the internal staff who get to share in the innovations created for clients and, to coin another term, 'the cobblers children will *finally* get new shoes'... because the shoes will have been designed for the client and, as we all know, there is nothing more enlightening than 'walking a mile in another person's (the clients) shoes' ;)

I hope our 'cobblers' in your organisation are up for it, there are a lot of new shoes to make ... welcome to the revolution ;) **1

**1 I had a choice between the song 'Children of the Revolution' by T-Rex or Tommy Lee Jones' quote from 'Under Siege' ... I went with the song :)