Drivel 2.0

Refactoring JQTI: JQTI+

2011-06-20T19:40:00.002+01:00

Last December I blogged that I was looking into maybe refactoring JQTI. Things went rather quiet after that point as my initial exploration was enough for me decide that this would be a lot of work and I found myself with plenty other stuff to do and not enough time to justify having fun of this nature (for some definition of the word "fun").

Since starting work on the JISC QTI Implementation and Profiling Support project a couple of months ago, I thought now might be a good time to look at this idea again and revisit my initial attempt with fresh eyes and with more experience of JQTI under my belt. My deeper knowledge of JQTI is both a blessing and a curse - I know much more about how it works, but I'm also now aware of a few other aspects of the software that are ripe for being refactored and improved.

So, what do I think can be improved? Here's a quick summary:

Splitting out state, logic and data: This is what I blogged about originally, and is still a very good idea. As I mentioned back then, it's not all that easy but I'm now convinced it's not only worth doing, but doable.
Stateless shuffling of interactions: For interactions that can be shuffled, JQTI implements the shuffling process by physically reordering the choices within the Object model. This is bad for two reasons. First, it's obviously not stateless! Secondly, it means that writing the assessmentItem Object back out as XML gives you something different from what you started with, which seems wrong.
Instantiating from XML: JQTI's hierarchy of Java Objects that mirror the concepts in the QTI spec all have a set of overloaded load() methods that instantiate Objects from XML in various forms. This is quite simple to use, but doesn't really mesh together too well. One problem is that resolving links to related resources, such as response processing templates and items referenced in tests only really works in some cases, and there's lots of duplicated code that doesn't handle all cases we'd need. Another issue is that JQTI keeps a deep copy of every DOM Node in the XML tree, which eats memory up for no great reason. JQTI also doesn't load in schemas (actually it now does in the fork I created for MathAssessEngine) which makes its validation a bit half-hearted, and adding performant support for using the core schemas is a bit of work that library clients have to do that would be better being done by JQTI. It also doesn't really "get" namespaces, but that's being a bit nitpicky! So, I envisage a refactored JQTI to have better support for locating and reading in XML, and controlling this process.
Better test navigation: JQTI has a concept of "Item Flow" that is used to help model the movement through a test. This idea works quite well in simple cases, but doesn't work so well for weird combinations of navigation and submission modes. So I think a lot of the test navigation ideas need refactored, which probably means I'll merge in the module previously known as JQTI-Controller back into JQTI as I'm not sure the additional level of flexibility that offered is much use.

So, there's even more to do than I first thought. I am clearly insane!

MathAssessEngine development update

2011-06-18T19:35:00.004+01:00

Following our successful award of JISC funding in the form of our QTI Implementation and Profiling Support project, I have now recommenced development work on MathAssessEngine. I'm about 2 months into development and things have gone rather well so far. I'm currently about to start the second big iteration of work so thought this would be a good time to summarise what work has been done so far, as I'm conscious of the fact that I've not been very good at blogging recently. (That said, when have I ever been any good at this?!)

Anyway, back to the point. There have been essentially 3 main strands of work going so far:

QTI 2.1 reference implementation: There has been a fair flurry of work and new interest in the QTI 2.1 spec as it advances towards completion. I've been using MathAssessEngine as a way of tracking new ideas and proposals as and when they've been mooted in order to let people play around with them and see how they work, so MathAssessEngine is playing the role of a reference implementation here, along with Graham Smith's JAssess. (That said, this is not a proper 100% implementation as it doesn't quite implement the whole spec and there are a growing list of historic bugs being noticed as well, but let's not talk about those today!)
Resolution of known issues: There have been a number of well-known issues with MathAssessEngine (and its parent QTIEngine) that have needed fixed for a while. One of the main issues is the testing functionality, which doesn't handle some of the more esoteric combinations of navigation and submission modes that QTI specifies. So this work is going to fix as many of these known issues as possible, which will probably result in significant changes to the way that tests work. (It's also bringing up new issues, but that's always to be expected!)
General improvements and refactoring: This is a good time to improve and tidy up the underlying architecture, based on lessons learned over the last couple of years and my growing experience of the inherited JQTI code base. I enjoy doing this kind of thing in a kind of masochistic way, but this practice invariably opens virtual cans of worms all over the place, which makes for a terrible mess for a while. So we'll have to see how far we go here.

So what are the key developments so far, then? Well, I'm quite pleased with what has been achieved so far. We're currently at the stage where individual assessment items render really nicely, with many improvements and better browser compatibility over the last version of MathAssessEngine. Here are some highlights:

Better MathML support: MathML is now rendered via MathJax, which means it will work really nicely in most modern browsers. The rendered content is now HTML5 (+ MathML) by default, though I still support XHTML1 (+ MathML).
Better QTI validation: All items and tests are now validated against the relevant schemas, as well as performing additional QTI-specific validation to ensure that they are going to work correctly.
Better input validation: Many of the interactions weren't validating responses correctly to ensure that they make sense, a problem that in some cases permeated all the way down into underlying the JQTI library. This has been resolved for all validations, and our fork of JQTI now validates all of the interactions it supports.
New rendering of mathEntryInteraction: Candidates can now input maths using an improved input widget that provides real-time AJAX feedback as to whether their input makes sense.
Improvements to certain interactions: The orderInteraction rendering has been completely revamped so that it can support the minChoices and maxChoices attributes. sliderInteraction has been redone to use pure JavaScript. The rendering for interactions has been improved with rewritten JavaScript and better handling of bad inputs.
Support for proposed new QTI expressions: We now support the proposed new mathOperator, mathConstant, statsOperator, min, max, lcm, gcd and roundTo expressions. The rounding logic in JQTI has also been rewritten so that it does the right thing when resolving tie breaks.
Better JQTI processing logic: The logic that controls the processing of assessment items has been improved to fix a number of known issues and support the spec better.

Phew! You may note from this that I haven't said anything about tests yet. Tests are going to be sorted out in the next development iteration. Because of this, they're not currently working in the latest snapshot of MathAssessEngine, which is why I haven't replaced the "live" version with the new one yet. But if you're feeling brave, you're welcome to have a play with the latest snapshot of development work at http://www2.ph.ed.ac.uk/MathAssessEngine-dev.

More updates later. I'll try not to take so long for the next update...

ASCIIMathParser.js released

2011-03-05T08:52:00.005Z

Peter Jipsen's ASCIIMathML script has been a very useful solution for getting maths on the web in some cases for a good few years now. In my opinion, one of its greatest assets is its very compact input syntax for maths, which is simpler and easier to pick up than LaTeX.

In the last few years, I have increasingly been using ASCIIMath purely as a means for parsing its input syntax into MathML, which is then passed to some other software component for further processing. For example, this idea is used in MathAssessEngine as a mechanism for students to input simple mathematical expressions as responses to assessment items. Other features in the script, such as pre-processing an HTML page, providing an online calculator widget, drawing graphics, and triggering the display of MathML in Firefox and Internet Explorer simply have not been used, so it has seemed a bit wasteful keeping them around.

In light of my very specific requirements for using ASCIIMath, I spent a few hours this week having a go at creating a cut-down version of the script that concentrates solely on converting ASCIIMath input syntax to MathML. This seemed quite an interesting thing to do for 2 main reasons:

I was keen to see just how small the actual parsing code is. (It is indeed very compact, which is nice, as it makes porting it to other languages seem like an easier option than it would have done looking at the "kitchen sink" approach of the original script.)
I wanted to minimise the dependency on browser-based JavaScript Objects to see if it could be ported to other JavaScript/ECMAScript environments. (I ended up just requiring a suitably decent DOM Document Object, which means the script could run on Java with the Rhino JS engine.)

The result of this experiment is a new script called ASCIIMathParser.js. I have posted this in the SnuggleTeX Math Playground and you are free to do whatever you like with it if you think it might be useful.

It is squarely aimed at developers like myself. There is an example of its usage in my new ASCIIMath input widget demo, and I have also done a very simple example showing it being used server-side in a Java servlet.

Implementation notes

I wanted to keep the structure of my script as close to the original script (v2.1) as possible so that I could keep it in sync with any future fixes/changes. To help me with this, I've put in lots of comments telling me where changes/cuts were made.

The first thing I did was find the actual entry point into the parsing code. This is the AMparseExpr() function. I then cut out the bits at the start of the script that handle the display of MathML, and the bits that provide the LaTeX, SVG and calculator functionality.

Next up was creating the new parser entry point, which is called parseASCIIMathInput().

I also had to replace the createMmlNode() with one which wasn't written for the Firefox vs. IE/MathPlayer dichotomy that ASCIIMath supports. There's still a dichotomy in the resulting function, since Microsoft's DOM still doesn't have a createElementNS() method, but the results are now "standard" DOM vs. Microsoft DOM, rather than Firefox vs. IE/MathPlayer.

I also removed the extraneous <mstyle> wrapping that ASCIIMath does, and commented out some bits of code and most of the old global variables that weren't used.

Finally, I wrapped the whole thing up in

function ASCIIMathParser(document) {
  ...

  this.parseASCIIMathInput = function(asciiMathInput) {
    ...
  }
}

This encapsulates all of the code into a simple class, and also demonstrates that the only real dependency is a suitable XML DOM Document Object.

It was quite easy to test that this code works in any modern browser. You really do need to create a new XML DOM to run the code since, if you're running in an HTML page, your document Object will be a bit deficient. But this is not hard, and I've provided a helper script that you can use for this.

I also spent an hour or so getting the script to run from Java using the Rhino JavaScript engine and Java's native org.w3c.dom.Document Object. This worked surprisingly well, with the exception that traversal of a org.w3.dom.NodeList via node[i] didn't work, so I did a search & replace to change these to node.item(i) in the code. There is probably a more elegant future solution to this.

Nevertheless, I am quite pleased at how easy this turned out. I haven't done the ASCIIMath LaTeX or SVG input, but that may well be possible too. (It really depends on how closely the code ties itself to browser JavaScript Objects.)

Feel free to play around with the code. Feedback, comments, bug reports are welcome.

New SnuggleTeX Math Playground

2011-03-04T19:17:00.004Z

Today I relaunched the "Math Playground" demo site that I originally created just before Christmas back in 2008. This site was initially created to try out and demonstrate some ideas for the JISC MathAssess project, which ended up seeding the SnuggleTeX Up-conversion/Semantic Enrichment process and the Jacomax project, as well as incubating some ideas that later found their way into MathAssessEngine.

This new version of the site has been rebranded as the "SnuggleTeX Math Playground" as one of its main purposes is to give me somewhere to incubate and refine some new ideas that might find their way into SnuggleTeX 1.3.0 or later.

The site launches with the following:

A purified version of ASCIIMathML called ASCIIMathParser.js that contains just the core ASCIIMath parsing code. This might open up new avenues for using and integrating ASCIIMathML in new situations. I plan to blog about this shortly...!
A trivial example demonstrating ASCIIMathParser.js running in a Java webapp.
A brand new version of the ASCIIMath input demo from the current SnuggleTeX website. This one uses MathJax for rendering the maths, enabling it to work in any modern browser. It also does real-time AJAX calls to a brand new SnuggleTeX up-conversion web service that lets users see whether their input "makes sense" while they type.
A version of the new ASCIMath input demo that uses SnuggleTeX's LaTeX as an input format, just for fun.

Have a play... you won't break anything. Probably...

SnuggleTeX Development Update

2011-02-26T17:27:00.002Z

It has now been a good few months since I released SnuggleTeX 1.2.2 way back in May 2010. Since then, I haven't really had as much time to allocate to it as I would prefer, but I have made some progress on putting together the next release, which will either be 1.2.3 or 1.3.0 depending on how things go.

Here's a quick run-down of recent developments:

Unicode input support: You can finally use arbitrary Unicode characters in your input as you can with modern LaTeX distributions. This still needs a bit of polish but the general idea is now there.
Simpler definition of many MATH commands: Many MATH mode commands are really just aliases for Unicode characters. Because of the newly added Unicode support, these commands are all now defined in simple text files that you can override by chucking alternative versions in the ClassPath, which will allow you to customise things easier than before.
Cleverer parser: LaTeX is a really complex beast to support as it allows you to use a rather messy mixture of markup styles. For example, the older style commands like \bf last until the next closing brace, whereas newer ones like \textbf{...} have a more tree-like approach that are easier to map to the XML way of doing things. SnuggleTeX has always been able to parse both, but was limited in how it would propagate style commands in the resulting output. However, I have now tweaked the core SnuggleTeX code so that things behave correctly. (This was actually quite a big change that required much chewing of fingers!)
Support for MathML 3.0: This isn't really as exciting as it sounds, but basically the MathML that SnuggleTeX generates is suitably unexciting that it's fine as MathML 2.0 or MathML 3.0. The unit tests now verify that the MathML generates is valid according to the latest RELAX NG schema for MathML 3.0, just to make sure I'm not giving you mince in your outputs.

Things currently in the pipeline:

Modern web page generation: Many of the options currently available for generating MathML-enabled web pages are getting a bit long in the tooth. The recent emergence of MathJax has made it far easier to create MathML-based web pages that will render in any modern browser, so I'm currently playing around with adding support for this. I also want to add support for HTML5 output which, when combined with MathJax, is something that is future-proof and already working in pretty much any modern browser as well. I'm still playing around with this, but something concrete should be committed soon...
New web demo applications: I want to update the demo web applications to try out some new stuff. This is currently happening in a little experimental spin-off webapp that I'll blog about very soon. Watch this space!
Rejigged website: I can never find anything in the current SnuggleTeX website. You probably can't either. I really need to fix this.

So there you go! The next release will probably be called 1.3.0 as some of the new features required a lot of changes to the internal API that some folks may be using, which warrants bumping up the minor version number. There may be a 1.2.3 bug-fix release that incorporates some minor bug-fixes I've done if an issue is discovered with 1.2.2 that would benefit from an immediate fix going out. We'll see...

Java and XML Gotchas #1: Don't pass a java.io.Reader to an XML parser

2010-12-06T19:30:00.023Z

This particular gotcha is a good example of a class of problems I call encoding mismatches and is an easy and rather nasty to trap to fall into for a number of reasons:

The Java API for XML processing lets you get away with it so effortlessly.
Novice Java programmers learn to use Reader classes to read in textual data and, "since XML is textual data", this seems an obvious and helpful thing thing to do when reading in XML.
English speakers don't usually notice anything is wrong until it's too late!

I have certainly made this mistake in the past myself, as have many people I've worked with over the years, so think it is definitely something that every Java programmer who uses XML should at least think about, even if you're already doing things correctly.

Before we even look at XML though, we need to take a few steps back and briefly remind ourselves of some of the things we should know about when communicating textual data.

Unicode and encodings

Every programmer needs to know at least a little bit about encodings, which are algorithms specifying how textual data should be represented as binary data for storage and transmission. Java and XML both support the Unicode standard, which defines well over 100000 characters and symbols in use throughout the world. In order to communicate all of these characters digitally, they need to be packed into bytes and, with a single byte capable of representing only 256 possible characters, this is clearly not a trivial task. One arguably old-fashioned approach is to throw away most of Unicode and select a subset that can be mapped into a single byte. This gives us popular encodings such as the 128-character ASCII encoding and the various Latin encodings, which flesh ASCII out with characters used in various European countries. More flexible encodings that allow you to use all of the Unicode character range necessarily use more than one byte to represent certain characters. Perhaps the most common example of these encodings is UTF-8, which uses between 1 and 4 bytes to encode each Unicode character and is very efficient for representing English text as it encodes ASCII characters in exactly the same way as the ASCII and Latin encodings.

Reading encoded text data

In order to read in textual data represented as bytes, you need to know the encoding that was used so that it can be decoded correctly. This simple fact often surprises novice programmers, who all too easily rely on things like a "default" encoding for reading and writing textual data. Default encodings are very convenient if you are the sole producer and consumer of your data, but they are are useless if you're using data you've obtained from someone or somewhere else. For this reason, most "protocols" that communicate textual data have a mechanism of telling you which encoding is being used so that you (or your software) can correctly decode it.

Encoding mismatches

If you read in textual data using the wrong encoding then you will find erroneous characters introduced into the decoded text and your users will start complaining about "funny symbols". You may also get an error report if you're lucky, depending on the API and the way the encoding algorithm works.

Noticing these mismatched encoding bugs can sometimes be harder than you imagine, especially when dealing with English text in the common encodings. As a Brit, the most common encodings I encounter are UTF-8 and Latin-1. Since ASCII characters are encoded into the same bytes when using the ASCII, Latin and UTF-8 encodings, encoding mismatches only become evident when using non-ASCII characters such as accented European characters or mathematical symbols, and it's not uncommon to encounter software that has managed to get into production without having thought of ever using such characters, leading to funny symbol reports from confused users. For example, a German ü character is encoded as 2 bytes in UTF-8. If these bytes are then (incorrectly) decoded using Latin-1, then you'll end up with 2 (wrong) characters instead of the ü. Fun!

Reading in text using Java

The simplest and traditional way of reading in textual data in Java is to use a subclass of Reader. You can correctly and consistently read a UTF-8-encoded text file in with the rather verbose:

InputStreamReader reader = new InputStreamReader(new FileInputStream(new File("myfile.txt")), "UTF-8");

Less experienced programmers might opt for the shorter:

FileReader reader  = new FileReader(new File("myfile.txt"));

The problem with this second form is that the encoding is not specified anywhere, so Java will use the "platform default" encoding, which may or may not be the correct one and will be specific to the computer the code is running on. (So, in particular, this form should never be used in "server-side" code.)

If you look at the java.io package Java API, you'll see that many Reader constructors let you specify the encoding that should be used, whereas many specify no encoding, using the platform default. This can be OK if you're reading and writing out text files locally, but you should only use these default encodings if you are 100% sure that the default encoding is the correct one, otherwise the character data will be decoded wrongly. Also, the Reader classes don't report decoding errors so it's hard to detect when things go wrong.

XML and encodings

The XML specification is clever here and allows you to specify the encoding within the (normally optional) XML declaration at the start of the file, using a default of UTF-8 if no declaration is found. Here's an example:

<?xml version="1.0" encoding="ISO-8859-1"?>

This says that the encoded binary representation of this XML file uses the ISO-8859-1 (a.k.a. Latin-1) encoding.

When you tell an XML parser to parse a binary stream, it looks at the first few bytes to work out which encoding should be used. It then decodes the stream using this encoding and parses the resulting textual data, hopefully correctly. Your XML parser is actually doing a lot of work for you here, which you should be thankful for. You should also let it do this work, as it's much more likely to do it correctly than you are!

If, on the other hand, you decide to decode your XML first (e.g. using a Java Reader class), then you need to know the correct encoding in advance. You'll then be passing character data to your XML parser and it will correctly ignore the encoding specified in the XML declaration since you have already decoded the text. If you decoded using the wrong encoding, then funny symbols will no doubt ensue.

Reading XML with Java

The Java XML APIs generally come with a number of overloaded methods for parsing, transforming and doing other exciting things to XML sources. Based on what you've read so far, you'll now generally know to avoid using ones that take Reader, favouring InputStream or File instead.

Here's an example of parsing a File with a SAX Parser:

public static void parseXMLGood(File file, DefaultHandler handler) throws Exception {
    SAXParser saxParser = SAXParserFactory.newInstance().newSAXParser();
    saxParser.parse(file, handler);
}

This is actually nice and simple in this case since the API helpfully provides a parse() method taking a File. In other cases, you might need to obtain a FileInputStream first.

Of course, as with all "rules", there are valid cases for breaking them. For example, if you've built up some XML programmatically as a big String, then using a StringReader is of course the right approach.

Conclusion... and moral of the story

Unless you have reason to do otherwise - and know what you're doing - you should always:

Pass raw binary streams (e.g. InputStream, File) to your XML parser
Let your XML parser do the decoding for you
Only use the Reader constructors that specify an explicit encoding

Refactoring JQTI

2010-12-01T07:15:00.008Z

During some fleeting moments of deep thought recently, I've started to think that the JQTI library might benefit significantly from a bit of refactoring to split out some of the competing types of information that it models.

What's JQTI again? Well, JQTI is a lovely little Java library that models the IMS QTI specification, which was created by the folks at the University of Southampton. When I say little, I should really say large, but necessarily so as the QTI specification is itself large and powerful. JQTI provides developers with a family of Java Classes that closely matches the concepts defined in the QTI specification. For example, there is an AssessmentItem class that mirrors the <assessmentitem> element in the QTI schema, which describes a single assessment item. An instance of this Class has methods to manipulate the actual data in the question (i.e. the XML attributes and element content), methods for validating data and inherited utility methods for reading and writing XML. But it also contains methods for doing the question, such as initialising template variables and processing responses and all of the question state is contained within the Object as well.

This self-contained approach has some nice qualities - it makes it quite easy to use the library and the API is correspondingly nice and simple too. However, there are some obvious issues with mixing things together in this way. First of all, imagine we're using JQTI in a testing system that issues the same question to 1000 students. We would then need to instantiate 1000 AssessmentItem instances: one for each student. Within these are 1000 identical copies of the underlying XML data, which is wasteful. Another issue with this model is with the statefulness of these JQTI Objects, meaning that we need to keep them alive for a long enough time, and there's no well-defined way of serializing these Objects into XML for passing around. This makes creating a RESTful API to JQTI-based systems a bit awkward. (Southampton now have a kind-of RESTful version of QTIEngine, but it is HTTP session-based so arguably not quite there.)

In light of these issues, I've spent a few days looking at the code to see how easy it might be to refactor things to split things up so as to separate (at least) the XML data parts, the candidate state parts, and the QTI business logic. In smaller models, this is not usually too hard, but it's actually going to be quite a bit of work for JQTI and I think it might be wise to make some compromises that would make code purists feel slightly ill. For example, one compromise might be to leave some of the business logic in the original JQTI classes, acting on passed state and context Objects rather than on underlying internal state. If I don't do this, then I fear we would end up having another hierarchy of logic classes as there's a lot of business logic in very specific places at the moment, and I'd then have to worry about weaving everything together. So we might have to be a bit pragmatic here.

Indeed, I have already tried bumping code around to split up the AssessmentItem part of the spec (i.e. the easy bit) to see how things might take shape and identify difficulties and risks with te process. Doing the same for AssessmentTest is an order of magnitude harder, though. And then there's the risk that I break more than I fix with this, and there's already limited unit test coverage so that won't provide much of a safety blanket. So I clearly need to think a bit harder.

So watch this space! For those that are interested, I already maintain a fork of JQTI in the FETLAR SVN repository on SourceForge.net. This JQTI-MathAssess fork adds in some hooks to make MathAssessEngine perform better with Maxima, fixes the (slightly bizarre... and arguably wrong) way that JQTI reads and writes XML and tracks the more experimental stuff that the QTI working group have been trying out as we move towards QTI 2.1. The refactored QTI stuff is going to be branched from this, but I won't give details until/if I commit to doing it.

SnuggleTeX 1.2.2 released

2010-05-26T16:20:00.003+01:00

Yesterday I slipped out another minor update release of SnuggleTeX. This release focuses mainly on bug fixes including, among other things, the addition of a safety net to prevent infinite loops when evaluating user-defined commands that inadvertently call back on themselves. You are therefore advised to upgrade to this version if possible!

I have also tidied up and improved the online demos slightly. One thing you may notice is that the ASCIIMathML demo now also shows the resulting MathML source while you type, which is quite fun, albeit in a hard to believe kind of way.

SnuggleTeX 1.2.2 release notes

SnuggleTeX home page and demos

SnuggleTeX project on SourceForge.net

Download SnuggleTeX

Announcing MathAssessEngine

2010-05-03T16:09:00.003+01:00

I'm pleased to announce the first public release of MathAssessEngine, a new open source engine for playing QTI 2.0 and 2.1 Assessment Items and Tests.

Based on the original QTIEngine code from Southampton, MathAssessEngine includes full support for the MathAssess QTI extensions that allow a Computer Algebra System (Maxima in our case) to be used within a question's logic, greatly increasing the usability of QTI in mathematical domains. MathAssessEngine also includes some additional minor features and optimised XSLT 2.0-based rendering that is easy to extend programmatically. It is also now easier for developers to integrate and use MathAssessEngine programmatically with other software components, such as our newly updated version of the "qtiplayr" plugin for Moodle that allows you to do full QTI 2.1-based assessment within Moodle.

This work was part of the JISC-funded MathAssess and FETLAR projects.

A demonstration version of MathAssessEngine can be found at:

http://www2.ph.ed.ac.uk/MathAssessEngine

which includes a number of sample items and tests for you to try out. You can also upload your own content, either from your local disks or directly from another website.

Further information on the FETLAR project can be found at:

http://aqurate.kingston.ac.uk/fetlar/

SnuggleTeX 1.2.1 is out

2010-04-03T15:12:00.002+01:00

A new release of SnuggleTeX (1.2.1) escaped into the wild yesterday. The observant among you may have noticed that I completely failed to anounce 1.2.0 when I released it back in February, so this blog post is my penance for it! If you're wondering what was in 1.2.0, have a look at my previous posting on 1.2.0-beta1, which ended up being pretty much the same as the final release.

This new version is mainly a bug-fix release to sort out a few obscure corner cases in the handling of mathematical "style" constructs like \textrm{...}. This also fixes one embarrassingly silly reversed logic case which would have prevented the example classes running properly. (Nobody reported that... eek!) And I've also added a little workaround to help people who might be affected by this bug in Firefox 3 that occurs when generating markup containing source annotations. (Interestingly, I've suddently started seeing activity on this bug after sitting dormant for a good year or so, so the lovely people working on this part of Mozilla may soon have a fix in the works.)

As with all releases of the form 1.2.x, this one is binary compatible with 1.2.0, so you should be able to simply replace your existing SnuggleTeX 1.2.0 JAR with this one, and I'd urge you to do so at your convenience. If you're using an older version of SnuggleTeX, have a look at the migration guide if you'd like to upgrade to 1.2.x.

Announcing Jacomax: a simple Java Connector for Maxima

2010-04-02T20:24:00.002+01:00

My recent(ish!) blog post on calling Maxima from Java drummed up a bit of interest from a few people who wanted to do this kind of thing, which was nice to hear about! And some people even managed to find the relevant code module buried within the QTItools SVN repository on SourceForge.net and get it running without any help from me, which was even nicer to hear!

As a result of this, I thought it might be useful to create a self-contained SourceForge.net project focusing specifically on what this code does and, in the last couple of months, have since completely rewritten the code to address some issues that my first draft had. (For example, you may have noticed deadlocks when sending lots of stuff to Maxima at once; the original code was really only written for doing comparatively trivial Maxima calls! This has all been rewritten and works very well in all of the situations I tested it on.)

If you've used the old code, you'll find the new code almost completely different to use, so you should consult the (mercifully short) set of documentation I've written for it. I think the new API is a lot saner than the old one, so it should feel more logical now. One other improvement in my new code is that it can try to guess how to call up Maxima by looking for it in standard places according to the OS you are using. This means that you will generally be able to run the code without having to supply an explicit MaximaConfiguration Object, though the old Properties-based configuration option is still around if you prefer it.

So, where is the new code then? You can find it in a new SourceForge.net project called Jacomax. As is customary for me, this project has a decidedly awful name, but it does make vague sense when you think about it.... JAva COnnector for MAXima! For convenience, I have put up an initial binary release to download (version 0.2.0), though you can also just check out the SVN as well if you want to stay bleeding edge. (To build from source, you'll need to install and know how to use Maven.)

I have tried to keep dependencies to a minimum here, but make a conscious choice to use a logging framework to make it easy to diagnose any issues with running the code on various combinations of Maxima, Lisp and OS platforms (which is actually quite scary when you think about it). I chose the slf4j logging framework - as I have been for most new projects - as it is arguably lighter, simpler and better behaved than Apache commons-logging. As a result, you'll need to know a tiny little bit about slf4j in order to make sure you have the right JARs in your ClassPath at runtime. In a nutshell, you need to make sure you always have slf4j-api, and then one other which depends on what kind of logging you want to do. If you download the Jacomax ZIP from SourceForge.net, you'll get two options here: either use the log4j (via slf4j-log4j) or do no logging at all (via slf4j-nop). And you're of course free to use any of the other logging options that slf4j supports as well. Other than logging, there are no other dependencies here. I've also bundled up some basic examples and a special JacomaxDiagnostic class, both in separate little JARs, which may be useful for some people. If you find that you can't get Jacomax to call up your Maxima correctly, you should run the JacomaxDiagnostic, which will generate lots and lots of logging information which should hopefully help work out where the problem lies.

SnuggleTeX 1.2.0-beta1 Released

2009-11-26T16:19:00.003Z

After some sporadic work over the last 5 months or so, I'm pleased to announce that the first beta release of SnuggleTeX 1.2.0 is now available to mess around with and/or download.

What's in this new release? Well, it pretty much incorporates everything I described in my recent SnuggleTeX Roadmap Upadate posting so I'm not going to repeat these details here as the last few weeks have mainly been concerned with tidying things up and filling in a few gaps.

If you've been using SnuggleTeX 1.1.0 (or even 1.0.0) and want to try out 1.2.0, I've written a short migration guide to help you incorporate the small number of API changes in this new release. I've also updated and improved the documentation though, as always, there's still a lot more work that can be done there...!!

In the mean time, have a play around with the updated semantic enrichment/up-conversion demos that show LaTeX being converted to Content MathML and Maxima notation. If you've tried this before, you'll maybe notice a new "Show Options" button that lets you tweak the up-conversion process a lot more than with SnuggleTeX 1.1.0. For example, you can now declare certain symbols to be functions, so that something like $f(x)$ gets interpreted as 'function f applied to x' instead of 'f multiplied by x'. More about this soon as I think it's quite funky, albeit in a rather geeky way.

Calling Maxima from Java

2009-11-21T10:28:00.003Z

One of the aims of the JISC MathAssess (and later FETLAR) project that I have been involved in is to investigate how to use (and slightly extend) the IMS QTI 2.1 draft specification to make it more useful in mathematical domains. One aspect of this has been trying to involve a Computer Algebra System in the question workflow, both in creating template variables (e.g. randomisation and avoiding degenerate cases) and in processing candidate responses (i.e. checking the students' answers).

The open source Maxima Computer Algebra System was chosen as an initial case, for various reasons. One challenge with this choice is that it is written in Lisp, whereas all of the other components in our QTI toolset have been written in Java. Therefore, an obvious problem to investigate was how best to call up Maxima from Java.

The question of calling up Maxima from Java has been asked before. For example, see post on the Maxima Mailing list from Maxima luminary Robert Dodier listing some possible solutions. In order to get something up and running very quickly, we opted for the first option of simply calling Maxima up as an external process and communicating with it through its input and output handles. This is quite a common initial strategy for these types of problems - I've done it before for LaTeX for example!

Calling up Maxima as an external process from Java is reasonably straight-forward, though slightly fiddly as it requires quite careful coordination between switching between "writing" and "reading" modes, otherwise the process effectively gets stuck. We also noticed that Maxima's input and output behaves slightly differently across different Lisp platforms (e.g. SBCL vs. CLISP), which we solved by cheating slightly with custom Maxima input and output labels and a bit of trickery to make sure that Maxima constucts like % (i.e. the last output) still behave in the correct way.

Following my initial attempt at this, I spent a few hours this week tidying this up and improving its test coverage and the means of configuring the process. (You have to tell the Java code where Maxima lives in order to make it start up correctly. And you may need to specify a bit more about its environment as well.) The code now currently lives as Maven sub-module called MathAssessTools-MaximaConnector inside the MathAssessTools module in the QTITools SVN repository on SourceForge.net.

Where next? Well, this might be something that other people are interested in so there may be mileage in turning it into a standalone open source project at some point. It's still very primitive in that it only really does what I need it to do. Things that are currently missing that might be useful include:

Pre-starting and/or pooling Maxima process for better performance. (Each fresh Maxima session currently starts a brand new Maxima process up there and then.)
Giving more control about loading in certain libraries when Maxima starts up. (Could some of these be contained in, say, a web application, where the notion of the library's location in the filesystem doesn't really exist? Maxima wouldn't like this very much!)
Adding in better support for converting Java Objects of certain types into equivalent Maxima forms. (We do this in elsewhere in QTITools in a rather specific fashion.)

Anyway, if anyone would be interested in this, let me know!

SnuggleTeX Roadmap Update

2009-11-07T19:57:00.002Z

It's been a few months now since I put out SnuggleTeX 1.1.0 and things have probably appeared rather quiet since then! My original plan was to put out a SnuggleTeX 1.1.1 release over the summer, which would fix the couple of bugs that were reported on SourceForge.net and add in some minor new features. As things turned out, I got a bit carried away with improvments and new features and it wasn't long before there were enough minor API changes to justify me making the next release 1.2.0, with 1.1.1 being an option if any show-stoppes were found with 1.1.0.

So... the next release of SnuggleTeX should be 1.2.0 and I'm hoping to get towards putting out a beta fairly soon. (You can play around with the SVN already, though some of the more cutting edge stuff has been changing daily so I'd suggest leaving it for a while. That said, the snuggletex-core module is quite stable now.)

What's going to be in 1.2.0? In snuggletex-core, I've enhanced the functionality for creating XML string outputs and web pages. This includes more control over serialization and support for generating named entities instead of numeric character references. (For example, ⁢ instead of ⁡. Note⁡ that this feature requires an XSLT 2.0 processor to be available as it's using the "character maps" feature that was introduced in XSLT 2.0. I've not decided whether to soldier on or throw an error if such a processor isn't available yet!) I have also replaced the old DefinitionMap class in snuggletex-core with something called SnugglePackage, which allows you to define you own "packages" of macros, environments, error codes and resource files for formatting error messages generated by your package.

I've also done a lot more work on the snuggletex-upconversion package, which attempts "semantic enrichment" on simple LaTeX inputs. This is still to be considered experimental in 1.2.0, though I hope some people might find it useful and any feedback would be appreciated. One new feature I've added is the ability to tell SnuggleTeX to make assumptions about certain symbols when being processed. For example, telling SnuggleTeX that 'f' is a function will cause it to handle 'f(x)' as a function application rather than a product. This mechanism also allows you to tell SnuggleTeX how to interpret brackets or various types, which is more flexible than my SnuggleTeX 1.1.0. These options can be set both within the Java API and within the LaTeX being parsed, which hopefully will make this quite easy to use. I won't say any more about this now as there's still work to be done, so watch this space!

More soon...

Let There Be Shite!

2008-08-06T21:34:00.003+01:00

Now that Leith no longer appears to be twinned with Rio de Janeiro, perhaps someone should come up with a new slogan/catchphrase/tagline for our lovely wee Toon.

Given the state of the streets these days, especially 'cos of the on-going tramworks and recent unusual spell of dry weather, I thought it would be fitting to celebrate our filthy pavements. Hence.... drum roll.... "Let There Be Shite!"

(That said, it's been chucking down all day so the pavements are unusually clean for a change. Even the blood stains and pools of vomit are starting to disappear...!)

SnuggleTeX 1.0.0-beta5 is finally released!

2008-08-01T20:09:00.004+01:00

I finally got SnuggleTeX 1.0.0-beta5 out the door earlier today after a flurry of activity over the last week or so. This is likely to be the final beta before I finally delcare that 1.0.0 is ready for the masses.

What's new in this release? Well, as far as the "core" part of SnuggleTeX goes (i.e. the LaTeX to XML DOM processing), this release contains little more than a few bug fixes and improvements. Outside this core, however, almost everything has been improved and tidied:

The web page generation has been improved to support another "page type" (a so-called UNIVERSAL_STYLESHEET output that generates an XML output using the MathML Universal StyleSheet to gain a good amount of portability, though losing a bit of speed and flexibility in the process). I've also bundled up the version of the USS that Keith did last year that adds support for Internet Explorer 7, which would be good to distribute.
I've added an experimental attempt at "down-converting" regions of MathML to plain old XHTML+CSS whenever possible. For example, something like a simple linear expression or a power can be marked up quite decently using XHTML. Everything deemed too complex is left as MathML. This is implemented as a rather simple little XSLT stylesheet. (Still XSLT 1.0 as I want SnuggleTeX to work with the bundled XSLT processor, though the documentation pages are using XSLT 2.0 via Saxon.)
I've migrated the feature I recently added to Aardvark that converts MathML to PNG images, via the JEuclid library. This has been deemed as an "extension" within SnuggleTeX, mainly because it sits outside the core functionality scope somewhat and also requires a shed-load of other libraries at run time. As a result, the Aardvark binaries are now distributed in two forms: one without and one with the extensions.
I've tidied up the demonstration webapp considerably, adding in Javadoc and splitting up and enhancing the existing documentation. I still need to write more docs, but this is a decent enough start I think.
Some of the main public classes have had their names changed for consistency. (This of course means that any existing code will need to change class names, but that's all that's needed so it's not too bad.)

What's still to be done for 1.0.0? Well, I'm currently planning:

Further enhancements on documentation and cosmetic changes to the website
Possible changes to MathMLWebPageOptions to split configuration into 2 layers: one lets you tinker with "raw" output options like whether to have XML declarations etc., the one above applies the existing "page types" as batch configurations to the raw options. This would make things more flexible for people who want to do weird things, whilst still keeping things simple for people who want to stick to the proven templates. I'm still not sure whether to do this now or keep it for 1.1.0...
Possible further package name changes to the internal classes as I don't like some of the existing names too much...

Once I've worked out how much of this I want to do, and then done it all, it'll be time for 1.0.0 to go live. Until then you can look at the new SnuggleTeX Website.

Bye bye TeX4ht!

2008-05-21T19:49:00.003+01:00

Today I finally said goodbye to TeX4ht as part of the codebase for Aardvark, replacing it with the new shiny home-grown SnuggleTeX. The refactoring wasn't really that bad and Aardvark is now a little bit simpler and easy to maintain, and feedback from the unit tests show that the validation process is now significantly faster.

Why ditch TeX4ht? Well, TeX4ht is a wonderful little tool and integrates very nicely with a TeX distribution to make it very easy to convert TeX-based markup into web pages of different flavours. On the other hand, there have always been some areas of disjunction in getting it to work with Aardvark:

It required a lot of configuration to make it work cleverly with "mixed markup". For Aardvark, this meant XHTML + MathML + Aardvark's own XML elements to represent structural entities like embeddings, key points etc. Problems could arise because TeX4ht appears to work by generating XML as text, meaning that we could sometimes end up with ill-formed XML. I got over this by a bit of hackery but it was unpleasant, brittle and required a lot of XSLT to fix up.
The outputs, while very nice, are geared towards turn-key deployment on the web. Making them a bit more "single-source" oriented was a lot of work.
Calling up a C-based tool from Java is never much fun. (Though I confess that I decided to preserve my sanity and forego any attempts at creating a JNI interface, using little shell scripts instead!)
The build process had to compile up and deploy TeX4ht since it's not quite ubiquituous. This obviously made the build process more complex and less appealing for new developers.
Trying to feed LaTeX errors back to users is not pleasant at the best of times, and became even harder when the errors might involve some bits and pieces that TeX4ht had inserted into the DVI file, which can be a bit too obscure for novice users. One of the design goals of SnuggleTeX was simpler and more accurate error reporting, which it so far seems to do quite well.

All of the existing unit tests have been updated in light of this change and everything is working well. I need to script up a few changes to existing content where it hits on some of SnuggleTeX's limitations, but otherwise the migration has proved surprisingly painless. Phew!

SnuggleTeX 1.0.0-beta2 Released

2008-05-20T13:51:00.003+01:00

Exactly one week after the beta1 release, it's time for another (beta) release to roll out the lastest bug fixes and a small number of new features. Release notes can be found at the new release notes page.

Work on beta3 is now starting in earnest (or "in vain", depending on your outlook on life). One new feature I'm thinking of migrating in is the recent stuff I did on using JEuclid to generate HTML+Image outputs in Aardvark (v0.14.0). This seems suitably useful and general that it should become a feature of SnuggleTeX, but at the same time I'm a wee bit worried that it might made SnuggleTeX a bit more complicated to use.

Talking of "how to use", I really need to improve the current (scant) documentation that is bundled in the distribution before we get to the final 1.0.0 release.

I'm still finding SourceForge.net's file release mechanism a bit ham-fisted. It's very 1998, darling!

SnuggleTeX 1.0.0-beta1 Released

2008-05-14T17:35:00.004+01:00

I've just released the first beta of SnuggleTeX 1.0.0 via SourceForge.net. It's a bit "beta" in more ways than one - this was my first attempt at using SourceForge's (decidedly clunky) release mechanism and there's not much in the way of useful documentation at the moment. For those reasons, I would probably recommend sticking with the SVN for the time being if you want to have a poke around as the source code (especially the "demo web app") contains some useful examples of using SnuggleTeX. I'm planning to write up something on the Wiki/documentation site in the near future as well.

The demonstration web app has been installed in a little corner of the School of Physics web server, which also doubles up as the project's official home page. Alas, it's far too minimal at the moment, but it does have a little form that you can enter some LaTeX into and see the resulting XHTML + MathML, which is mildly amusing, if only for a couple of nanoseconds.

I've now moved on to a bit of necessary surgery: grafting SnuggleTeX into Aardvark as a replacement for the old tex4ht solution. Things are currently quite bloody, though it's proceeding well and is actually proving rather enjoyable so far. Once completed, a useful by-product of this process will be a number of bug-fixes and minor enhancements that can be fed back into SnuggleTeX in time for the proper release of 1.0.0, which I'm hoping will occur by the end of June.

More progress later... I think I've left some blood-soaked swabs inside poor old Aardvark!

SnuggleTeX is born!

2008-04-16T18:58:00.003+01:00

After a couple of months of planning and development, my new system for converting the sort of simple LaTeX we use in Aardvark to XML is beginning to take shape. It's even got a name now - SnuggleTeX. Why's it called that, I (don't) hear you ask? Well, firstly I don't like acronyms. Secondly, TeX is already a bit cryptic sounding so I wanted something cute to calm it all down a bit. My first idea was originally going to be "snuffle", since the main motivator for the new library is Aardvark and I have this mental image of aardvarks making a kind of snuffling sound when they, erm, aardvark around. But I decided when lying in bed one night that "snuggle" sounded cuter so the name stuck. To ensure that I don't change my mind, I've already created a project page on Sourceforge (see http://sourceforge.net/projects/snuggletex/) which, at the time of going to press, contains virtually no useful information about the project.

So what does SnuggleTeX do? Well its main development goals were driven by the requirements of Aardvark and some of the experiences gained from the existing TeX->XML engine that Aardvark uses:

Create a 100% Java library for converting a reasonable subset of paper-agnostic LaTeX into XML.
Output XML should be XHTML + MathML, additionally allowing custom XML elements to be created in the output.
Support \newcommand, \newenvironment and friends.
Provide various modes for error-handling (e.g. fail, ignore, log error but keep going, embed error message into XML, ...)
Output useful error messages, making it easy to locate errors.
Minimal (currently no) dependencies on other Java libraries.
Add optional support for inferring semantics in certain Math expressions
Be amenable to be plugged into things like Wiki engines (Confluence perhaps?!)

As of writing, SnuggleTeX is still in my local CVS repository but is now nearing the point where it can be imported into Sourceforge's SVN repository. (I will give the project layout a bit of a much-needed tidy first.) It supports about 90% of the macros I want to support and I have a test suite of around 250 (and still growing) functional test cases, which have proved very useful for minimising regressions issues. I am currently running it against a set of pages from Aardvark to check for issues and compatibility with existing content and am working out a migration plan for integrating it with Aardvark.

Performance is reasonably good at the moment, though I've identified some potential bottlenecks when processing large documents that can be optimised in due course. One of the things that makes tokenising LaTeX quite hard is the way user-defined commands and environments work. They essentially act as text substitutions, which means the "working view" of the input changes during tokenisation and keeping track of where each character belongs with respect to the original input requires a wee bit of work.

Moodle Quiz XML support in Aardvark

2008-03-14T13:28:00.005Z

Earlier this week I installed Moodle on the laptop to see how easy it would be to export Aardvark MCQs from Aardvark (and hence DUMP) into Moodle, which would be a useful new addition to the export formats currently offered.

After playing around for a bit, I quickly had my hands on some samples of Moodle's Quiz XML and found some (rather scant) documentation on the XML on the Moodle documentation site.

Initial impressions are interesting. The Moodle Quiz XML is generally nice and simple, and not too far removed from Aardvark's own XML schema for MCQs. This, in itself, is quite a refreshing change from all of the QTI stuff I've been working on recently.

On the other hand, there are a few strangely smelly bits. First of all, there doesn't appear to be a schema or DTD anywhere for the XML, so I've had to rely on wiki-ish (and hence not quite 100% complete) documentation and a bit of poking around in the source code. Secondly, all of the XHTML content in the Aardvark MCQs has to be specified as CDATA-escaped XHTML within Moodle's <text/> elements, which is similar to how QTI 1.2 does things but is not that great and is easily abused. Also, there are 'format="html"' attributes that are not consistently used across the various elements. Handling of images is a bit strange too. There is a single <image/> element allowing a single image to be specified. Poking in the source code demonstrated that this must be accompanied by a <image_base64> element containing the actual image data (encoded in Base 64), whereas the documentation claimed this was optional.

So how easily do the Aardvark MCQs fit? Not too bad, actually. For historical reasons (and I suspect this is the same with Moodle as well), Aardvark MCQs allow a "question image" to be specified as well so it's "just" a matter of embedding this into the XML. However, we also allow arbitrary images to be linked in and embedded from within the various blocks of text in the question. These inline images are a problem for Moodle. When doing the QTI and Respondus exports, I simply remapped these links and bundled up the image code inside the resulting ZIP files but, since Moodle just essentially outputs the CDATA-escaped XHTML fragments as-is, these relative image URLs end up being resolved against the wrong base URL (one of the PHP scripts controlling the question display), which don't work. That leaves me with 3 options:

Use absolute links back to read the images from "our" server.
Get the end user to specify the base URL of their Moodle instance and have the image links rewritten to point to that server when exported.
Try to insert some fancy-pants JavaScript into the XHTML fragments that will dynamically rewrite image links so that they resolve against the appropriate Moodle PHP script at run-time.

None of these are particularly attractive. Option 1 requires us to ensure we never delete content or move things around, and also works kind of orthogonal to the existing access control model that Aardvark has. Option 2 is a bit of effort for the end user, it depends on Moodle not changing the way it organises files in future and the resulting exports are not exactly very portable in this case. Option 3 might just work but is probably too clever for its own good and would also make the questions hard to re-edit.

How many images do we have? Well, as we can't realistically use MathML in these exports (since there's no way of getting an out-of-the-box Moodle to trigger the appropriate "slap in the face" that certain browsers need to display it) we'd normally have to export Mathematics as images, just like we do in some of the QTI exports. That potentially means a fair few images to support. But, one nice optional feature that comes with Moodle is a nice wee filter to automatically convert TeX fragments to MathML - provided of course there's a working TeX distribution on the Moodle server - which is something that we can exploit quite easily. So, stepping back for a minute, this image problem won't really bite us that often at all as we don't have all that many inline images in our questions, save for a few graph-based questions.

How are things coming so far? Well, I've got a working XSLT-based pipeline to generate the required Moodle XML and do the Base 64 encoding and link remapping as steps in the pipeline. That's all working quite well, I just need to finalise my decision on what to do with the images and that'll be it ready for more testing....

Aardvark goes into Sourceforge

2008-03-13T21:16:00.003Z

A couple of weeks ago we decided to open-source Aardvark, one of the projects I find myself working on quite a lot of the time at work. I've been working on this on and off for the last 5 years or so, so it's become my little baby in some ways! Anyway, it's been accepted into Sourceforge as AardvarkCMS (someone's already got a project called Aardvark - the swines!) and I've just finished migrating its existing CVS repository over to a spanking new Subversion repository. The new project page is still a bit bare - there's no project web page as yet and I haven't packaged up the current release for download but I plan to get there soon.

Initial impressions are mixed. Having a remotely-hosted SVN project is much, much slower than our little local CVS repository and I'm experiencing a few strange quirks with Subclipse, the SVN plugin for Eclipse. (For example, shared project launchers don't appear to be working with SVN projects, though I'm not sure why. And it's taking forever to synchronise the repository at the moment - which is very annoying as that happens after every commit.) I'm in the middle of a small refactoring at the moment but will maybe try checking it all out again once I've committed the latest changes.

Hot on the heels of Aardvark will hopefully be my new Java-based LaTeX(ish) to MathML converter, which is coming along nicely but not quite ready for public consumption. This will hopefully replace the existing TeX4ht solution that Aardvark uses, simplifying the build process considerably and making it a more attractive proposition for people who want to download and install it.

I'm also planning some minor refactoring of some bits of Aardvark, essentially to tidy up some of the most recent new features on QTI/IMS exports and stuff. Some of these refactorings will go all the way down to the library of SAX utilities that Aardvark uses, which is due for a bit of work. All of this is a bit nicer now that we've moved into SVN, even if it's bloody slow at the moment!

I'm also thinking about getting us a space on the University Wiki for our e-Learning projects. This would be a nice place to consolidate the existing (and often scant) documentation on Aardvark's internals...

Auld claes an' porage!

2008-01-06T21:05:00.001Z

Big Wheel on Princes Street
Originally uploaded by Dave McKain

As the sun sets on the Edinburgh Christmas, it's time to get back to normal - "auld claes an' porage" as my Mum always says. To be honest, I'm quite happy to be buckling down to the familiar old routine after the break: I've eaten far too much cake, my sleep currently doesn't have any discernable pattern and the hoose is still bloody freezing following the sad but inevitable death of my boiler on Hogmanay!

First Snow of 2008

2008-01-03T23:03:00.001Z

First Snow of 2008
Originally uploaded by Dave McKain

Well, today brought the first few flurries of snow over Edinburgh for 2008. In typical fashion, the snow in Leith was a bit wet and didn't stick around for very long but it was nice to see after such a long absence.

As luck would have it, my boiler decided to give up the ghost on Hogmanay so it hasn't been very warm inside these last few days. Luckily, I persevered with trying to light my rather elusive living flame gas fire and I've now got a rather toasty living room, with the oven nicely keeping the kitchen warm. A diet of curries has also help with hot air, though it doesn't always smell as nice as it could do. It's also slightly annoying that a new boiler is going to cost £LOTS, and that central heating tradesmen don't appear to be in a rush to help sell me one at the moment. Ho hum!

2008 has been a funny year so far. I'm not sure I'm a big fan as yet but it must surely get better...

Tonight's DJ Playlist

2007-09-29T01:06:00.000+01:00

Tonight I did another DJ set in support of Rich's Band (The Remnant Kings). Funnily enough, minor Celebrity Dominik Diamond's new band was on the line-up! Here's my playlist:

Sonic Youth - Teen Age Riot
Primal Scream - Blood Money
The Fall - A Past Gone Mad
We Say Party! You Say Die! - The Gap
Klaxons - It's Not Over Yet
The Jesus And Mary Chain - Never Understand
Nine Inch Nails - Survilalism (Tardusted)
Goldfrapp - Ride A White Horse
Muse - Starlight
The Strokes - Hard To Explain
New Order - Temptation (Secret Machines Remix)
Blondie - Atomic
Madness - Baggy Trousers
New Young Pony Club - Ice Cream
Presets - I Go Hard, I Go Home
Amy Winehouse - Back To Black
The Smiths - The Boy with the Thorn in his Side
Electric 6 - Danger! High Voltage
Big Black - Columbian Necktie
Primal Scream - Loaded
Joy Division - Love Will Tear Us Apart
Madonna - Deeper and Deeper
Nelly Furtado - Promiscuous
Linkin Park - What I've Done
CSS - Let's Make Love and Listen To Death From Above
Buck's Fizz - Making Your Mind Up

One Of Those Days...

2007-09-19T22:39:00.001+01:00

Clever old me. Tonight I went to the gym after a stressful day at work, took my trainers off and stripped off, just as I would normally. However, I then realised that I'd forgotten to bring my kit with me, as was confirmed by a frenzied ferret around inside of my backpack. Pausing for a moment to try to think of an appropriate emergency changing room protocol, I dashed off to the toilet to waste a bit of time washing my hands before calmy putting my clothes back on, pretending to look suitably worked-out, then walking out. In a pathetic attempt at disguising the fact that I'd only been there for 5 minutes, I left my jacket off hoping that I'd look different to the desk staff on the way out. Twat.

To add insult to injury, Tesco are currently doing 50% off on these rather addictive fortified chocolate milk shakes that I've managed to convince myself are healthy, even though they're most likely not. Walking past the shelf whilst pretending I was actually there to buy healthy things, I counted 5 bottles. But just as I turned back to pick some up, this bloke coolly flounced in from nowhere and swept the whole lot into his basket. If that person was you, I hope you spilled it down your trousers in the street, leaving you with a questionable looking stain.

What a fecking day!

Bad Shoes Horror

2007-09-17T21:28:00.000+01:00

It's been one of those days. After waking an hour before the alarm, then falling back asleep only to dream that I was still awake, I decided to go down to the gym to attempt to levitate weights by looking at them a bit funny, hoping it would lead to a vague feeling of motivation and contentness. (More accurately, it was an attempt at procrastinating as I had a 1500 conference presentation abstract to write so was looking for an excuse to rearrange my working day to make that seem less painful.)

Normally this works quite well, but today I was faced with a hellish affront that I had not prepared for. Someone was wearing Very Bad Shoes. In fact, I'd go as far to describe them as "offensive". Even worse, we was patently pleased with the fact and seemed to be aware of my obvious disgust as every time I opened my eyes, he'd be sitting on the machine opposite me.

So what were these shoes? Well, for once they weren't Crocs. (Don't get me started on them... that's for another day. But seriously, Crocs... WTF?!) No, he was wearing a pair of those old-fashioned leather shoes that make a pathetically inane clackety sound as they limp around. The sort of things a precociously annoying public school child might have deserved to wear in the olden days, just because they're precociously annoying. But this guy was too old for that. To compliment these travesties, he was wearing specially selected white Terry toweling socks with them. As far as I am aware, that has not been considered acceptable since 1985.

I should probably be concerned that I can be so badly traumatised by something so inanely trivial...

Top Bar-Clearing Songs

2007-07-20T22:20:00.000+01:00

A few months ago when doing one of my (admittedly crap) DJ'ing stints for some friends of mine, I quickly realised that even though chucking out time was fast approaching, a hardcore lump of audience evidently had no plans to leave. "Bollocks to that", thought I, "I need my bed!". After pondering for a few moments, it dawned on me that I should use my position as keeper of mixing desk to try to make everyone leave by playing music carefully crafted to empty the bar as quickly as possible.

My first idea was to embarrass them all by playing music they'd consider to be uncool. The remains of the audience at this point consisted of people just pushing 20 with an apparent penchant for The Fratellis and other similarly unkempt smelly bands with unacceptably bad hairstyles. Luckily, help was at hand - someone had left a "Christmas Party Mix" CD on the mixing desk. And a particularly cringeworthy CD it was too. (It wasn't mine, honest!) After a quick scan, I chose the old wedding reception favourite Agadoo. Surely they'd hate it, I thought, they're too young to see the kitsch value and are probably embarrassed that their parents relish dancing to it with tongues firmly placed in cheeks and nonchalant looks of knowing post-modernism.

Bizarrely, my plan failed miserably. People started dancing. "Bah!", I thought. But it was quite fun, so I put on "Can We Fix It?" by Bob the Builder next as that's one of my guilty pleasures. (Don't tell anyone. Nobody reads this anyway...)

After ploughing through "Making Your Mind Up" by Bucks Fizz, "Barbie Girl" by Aqua and "Last Christmas" by Wham, I realised that I obviously needed to put more thought into the art of selecting Bar Clearing songs in future and ended up just turning the mixing desk off and going home. How very rock and roll.

Now, being an analytical old fruit, I've since pondered a bit more on what makes a good bar-clearer. One obvious first idea that springs to most people's minds is to choose something heavy, loud or shouty. But I'm not convinced. Actually, I find music like that to be quite fun and would probably start jumping up and down to it and wouldn't be surprised if others did too. However, one possible example I did think of is that old chestnut "Discipline" by Throbbing Gristle:

I like this because Genesis P-Orridge's out-of-time, out-of-tune and generally out-of-everything shouting is particularly grating to most people, and the accompanying music offers little in the way of hooks for boozed-up teens to get caught on. Will it work? I dunno!

My next idea is to choose something from a genre that the target audience probably despise. Given the plethora of drainpipe trousers and aura of "this is real music" that permeated that evening, I thought an appropriate sub-genre of dance music might suffice. Now, your average The View or The Kooks fan probably likes a bit of a dance now and again so choosing the correct sub-genre is vitally important. After a bit of thought, I chose "Gantz Graf" by Autchre, an example of so-called "Intelligent Dance Music":

Why that? Well, even though it's theoretically "dance music", in practice it's neither of those: there's actually no melody whatsoever and you couldn't possibly dance to it! Which means that they won't have heard it in a club before so it won't bring back heady memories of boozed up shenanigans and the distant smell of vomit. Surely it would go down really badly? But what if some of the them were like me? I'd like it! Hmmm...

The last of my current burst of ideas was to pick something that's more-than-a-litte-bit out there, preferably something that'll scare the hell out of them. That's where a bit of carefully chosen Nurse With Wound often comes in handy. For example:

This one's perhaps not the best example but it's one of the few NWW "videos" that I could find to put on here. A better selection might be the catchily-titled "Nana Or A Thing Of Uncommon Nonsense", which drags the old favourite "Na Na Hey Hey Goodbye" through several hedges backwards through a snow-storm uphill both ways. And very spiky hedges at that.

Which one will I use next time? I dunno yet. Any further ideas?

Party Like It's 1982

2007-06-06T20:21:00.000+01:00

Well, since it's not my birthday and I'm also not 25, I thought that would be a perfect excuse to look back at some great music that came out in 1982!

Temptation - New Order

This is probably my favourite New Order single and one of my all time favourite songs as well. Their first self-produced effort after dropping Martin Hannett from production duties, this is wonderfully shambolic fun. Actually, the first 30 seconds of the 7" mix are so exciting that I always have to listen to the opening twice. I usually to prefer the 7" mix of this (pictured) as it has a more synthy, rigid feel than the 12", which is a bit looser and more meandering but they're both classics, as is the brutal B-side "Hurt". Being typical New Order, these original mixes of Temptation aren't as well known as they should be as they were hard to get hold of on CD for a long time - the version on Substance is the re-recorded 1987 version which is good but lacks the demented nature of the original. Listening to this still evokes heady feelings of being pelted with bits of plastic... it's aged very well.

Party Fears Two - Associates

Another song for which the word "bonkers" was invented, this showcases the multi-octave operatic voice of Billy Mackenzie and is probably the most well-known of all of the Associates' singles. This song has always managed to feel both timeless and dated at the same time. A word of friendly advice: never attempt to sing this at Karaoke; you'll never do it justice. People will laugh at you. Don't do it!

Golden Brown - The Stranglers

I remember being fascinated with the video for this when it came out in 1982 and the song is equally mesmerising, with its combination of 3/4 and 4/4 time signatures and almost trancey ending that works well when driving into the sunset. (Surprising, given that I don't have a car!) The clever bastards.

New Gold Dream - Simple Minds

This is my favourite Simple Minds album which, in my opinion, perfects the very "European" sound that they had been developing over previous the 3 years or so before they started moving towards a more "American", commercial sound. The feel of this album is warmer and more colourful than its predecessors, showcasing an impressive blend of electronics and more traditional guitar-based instrumentation that still doesn't get the recognition it deserves. (That said, songs from this era are still popular amongst DJs). An under-rated classic - don't let memories of Jim Kerr later looking windswept in big shoulder-pads put you off!

Today - Talk Talk

The best song from their first album, which got Talk Talk lumped in with other synthpop and Duran Duran wannabes, this still sounds fresh and vital today. And Talk Talk later proved early detractors wrong with their gradual progression into uncharted, unique territories, producing some of the most beautiful music of the late 80's.

Pornography - The Cure

Ah, this brings back memories of being suitably angst-ridden as a student. Of course, I was only 8 when this came out and probably would have run a mile if I'd seen something like The Hanging Garden on the telly, though I do remember liking The Love Cats when it came out the following year. This is the sound of the Cure going a bit loopy, it bludgeons you into submission with slabs of drums, doomy electronics and spiky guitars. "I must fight this sickness, find a cure" wails our Bob at the end, and I don't think anyone would disagree. Intriguingly, the cure came in the form of a bizarre move towards a more commercial sound with the inspired follow-up single "Let's Go To Bed". Hmmm... I must dig that out again too.

Leave In Silence - Depeche Mode

Even though this is perhaps not one of Depeche's best singles, it does demonstrate them fleshing out a blueprint for their future sound after their early forays into frothy pop (New Life, Just Can't Get Enough) and the departure of principal songwriter Vince Clarke. It's also their first Mute single to have a 'BONG' catalogue number prefix (to date, there have been 38 more since then!). It's still a good single, and one of the best tracks off the uneven "A Broken Frame" album that followed shortly after. It also showcases some of the nice counterpoint vocal harmonies that DM used to do, which never really got enough recognition.

The Dreaming - Kate Bush

This is the sound of Kate going a bit mad, and all the better for it too. There's no point trying to describe this aside from saying it ends with her doing donkey impressions - just give it a listen some time. Come to think of it, I really must have a go at "Sat In Your Lap" at the karaoke some time. Death by karaoke... sounds good!

Big Mess - Devo

The opening synth line of this one inspired the Kaiser Chiefs' "Every Day I Love You Less and Less", which possibly explains why I don't hate that song as much as everything else the KC's do. Big Mess is simply good fun synthpop. It's also great listen to at the gym for some reason.

Homotopy To Marie - Nurse With Wound

Ah, a nice difficult one to end with. This is perhaps my favourite out of the early NWW canon and feels more focused and consistent than their previous LPs. Like a lot of their music at this time, this is very close to "Musique Concrète" and has a very dark, psychedelic atmosphere. Not one to play at parties, unless you hate your guests! Hmmm... my birthday party is coming up soon...

Radio Hell

2007-06-06T20:06:00.000+01:00

I've long been convinced that my brain occasionally finds itself unwittingly tuning in to music being broadcast from a dark unknown. A parallel universe. A place of fear, dread and misery. No, not Kirkcaldy, but Radio Hell.

<Insert doomy laugh here/>

Today was no exception. People burning in hell today were apparently subjected to the intriguingly dubious tones of the following lost gems on infinite repeat:

The theme tune to Wogan (the really awful later version with unfortunate saxophone smorgasbord)
The theme tune to Bod
"The Lion Sleeps Tonite" by Tight Fit
"So Macho" by Sinitta
"Boys Boys Boys" by Sabrina
"Riding Along in my Automobile" as covered by Status Quo

Repent! Or you too will have to endure this after your demise!

I think this is pay-back for last Friday night... I'm going to clean the shower curtains and anoint myself with green tea and Deep Heat to try to make up for it.

The Psychic Toilet Unblocker

2007-05-28T19:24:00.000+01:00

Some bizarre news from back home in Fife...

Local lass Sapphire Lumsden, 21, is to appear in the new series of Most Haunted when filming shoots later in the Summer.

"My funny uncle used to keep this big pink rucksack in his wardrobe" said Sapphire. "One day when I was wee, I had a look inside and this big wobbly rubber thing fell out and bounced onto the floor, like a startled fish. But my Uncle caught me and gave me a right telling off. He said it was a decorative toilet unblocker that had been in the family for years and could communicate with spirit beings and that I must never ever go back into the rucksack as all sorts of stuff could happen and my legs might fall off. I was too scared to do that again."

But tragedy later struck. Sapphire went to visit her Uncle Leroy one Sunday morning in the remote Fife fishing village of Methil, only to find him lying in a pool of blood. "The police later told me he'd been bludgeoned to death with the toilet unblocker." It was only then that Sapphire discovered the powers of the plunger. "They gave me the plunger back and when I picked it up, it started to wobble all over the place. I couldn't believe it! My Uncle was trying to communicate with me through the plunger." She promptly arranged a seance with her friends and performed a special ouija board ritual with the plunger. "We were all packing ourselves at first, but knew what to do from the telly. I asked him all sorts of yes/no things and the big rubber plunger wobbled the right way every time. I even asked him who I was going to marry and it spelled out 'Fljib'. I've started searching the Fife phonebook for him. He sounds like he's maybe foreign - Wales or something - only 1039 more pages to go...."

The plunger has since caused a stir throughout sleepy Methil and the surrounding goitre. People have even come from afar as Leven and Buckhaven to witness the spectacle, now the focal point of a touching shrine in Sapphire's front window. It's only after appearing on local radio that it attracted the gaze of the Most Haunted crew. "It's great", exclaimed Sapphire, "I'm going to be on the telly - I'll have to get a new outfit from New Look first!". Yvette Fielding is said to be "excited" at the prospect of getting her hands on the plunger.

Sweet Dreams (aren't made of that)

2007-05-28T08:42:00.000+01:00

I normally have the most boring dreams imaginable. They generally involve public transport of some sort (plains, trains, buses) misbehaving in some way (breaking down, going the wrong way, not turning up), resulting in some mild terror (me being late for something really important, losing my shoes, forgetting my sandwiches). So it was quite a change to dream of something different last night, though, with hindsight, nothing actually made much sense.

First thing I remember was being at a conference and not having a clue what it was about. I've actually felt that one before in real life so that wasn't too strange in itself. But part of the conference involved being on this new game show where they ask questions that are so difficult that nobody ever gets any right. In fact, it's so hard that everyone ends up owing money to the game show by the end of it. None of the questions made any sense and I ended up losing a lot of money. Poor Joan Rivers was so petrified by it that she downed a whole bottle of vodka and ended up making a bit of a scene.

After losing all that money, I decided to buy a posh car (a Jaguar, I think) over the internet. I'm not sure how I was planning to pay for it but I must have seen sense as I decided to take the bus instead when I went to Kirkcaldy to pick it up. Aargh! Public transport again. Reflecting back, the concept of Jaguars being sold in Kirkcaldy is as daft as the rest of the dream...

The worst bit was later catching myself in a mirror and realising that my chest hair had mostly fallen out. This was horrifying for two reasons: first, my only tangible bear credentials were in tatters and, secondly, I'd be hoovering up lots of curly hairs for the next 2 weeks. This was too much to take so I did my emergency wake-up routine, went for a glass of water, (checked that I really did still have chest hair) and went back to sleep, hoping for a nicer dream about missing a bus. It never came. (The dream, that is. Not sure about the bus...)

fac51 - The Haçienda

2007-05-10T20:32:00.001+01:00

We all went down to Manchester for GBBBBBBBBBB10 (or whatever it was called) last weekend and I accidentally wandered past the old Haçienda building down near the Deansgate. It's been turned into posh apartments and offices now but it was still instantly recognisable and brought a wee tear to my eye. Sniff! I want one of these signs for the hall, even though it really won't go with my decor... I supposed I could redecorate for it!

As fate would have it, Peter Hook announced that New Order had split up a couple of days after the visit. That also brought a tear to my eye, but for different reasons. I'm just glad we managed to see them do such a blinding set when they played in Glasgow last November. I never thought I'd ever hear them tear through Bizarre Love Triangle, Temptation, Blue Monday and Perfect Kiss but they did and, for that, I'll die a reasonably happy bunny.

RIP New Order