Pygmentising Hakyll's Syntax Highlighting

Contents

• The problem
• Playing with pygmentize
• Integration
  • Pandoc
  • Hakyll
• Conclusion

By default, Hakyll uses pandoc to generate syntax highlighting for all kinds of different programming languages. However, even in simple examples the html this produces is unsatisfactory. Thankfully, the two programs are almost infinitely customisable, and changing pretty much any setting doesn’t usually involve a lot of work—this is no exception. Using pygmentize as an example, I will show you how you can swap out pandoc’s native syntax highlighting with pretty much any third party tool that can output html.

The problem¶

Pandoc uses the skylighting library to generate syntax highlighting for a given block of code. Skylighting, in turn, uses kde xml syntax definitions for the respective tokenisers. However, even for simple examples I don’t agree with the html this generates. Consider the following Haskell code block.

Pandoc would generate something like the following:

One can already see a few things wrong with this: (i) in the type signature, the name of the list is smushed together with the separating double colon (worse: it’s just in the “other” syntax class), (ii) in the actual definition, fibs isn’t assigned any class at all, and (iii) the assignment operator is also in the “other” class, instead of something related to it being a built in operator! As one can imagine, this only gets worse as snippets get more complicated.

These kinds of issues, combined with the fact that certain languages—like Emacs Lisp—don’t have any syntax definitions at all, annoyed me enough to look for an alternative way to highlight code on this website. There are of course many options to choose from; I went with pygmentize, solely because I already had it installed. All that’s left is to tell pandoc and Hakyll to make use of it. As mentioned, this thankfully doesn’t turn out to be very difficult!

Playing with pygmentize¶

Having never used pygmentize as a command line utility, I expected this to take some work—possibly involving Python shudder—but all of the necessary pieces are already present in the cli. First up, the -f option specifies the formatter to use, which will decide the shape of the output.

We can test how this highlighting looks straight away; executing

produces an html output along the lines of

This looks much better! The class names are kind of obtuse, but pygmentize can also give you nicely annotated css styles for its supported colour schemes. For example, the following is a small excerpt of the output:

You can redirect this into a pygments.css file, link to it (e.g., from your default.html template), and be on your way. The annotation also makes it very easy to change that file after the fact, in case pygmentize does not have the theme that you want.

Integration¶

The idea of what we want to do is quite simple: for every code block in a given post, shell out to pygmentize, and use its output to replace the block, somehow making sure pandoc doesn’t touch it afterwards. Let’s solve this step by step.

Pandoc¶

Pandoc has an aptly named Pandoc type, which represents the internal structure of a document.

We’ll neglect the metadata for now and just look at the Blocks; specifically, we want to zoom in on two constructors that will give you everything we need:

To get a feeling for how these CodeBlocks look, again consider our fibs example from above. By default, the corresponding CodeBlock for this would look something like

Importantly, the language (if any) is the first argument of the classes field of Attr.

A strategy begins to form: look for all occurences of a CodeBlock in the Pandoc type, and replace it with a RawBlock "html" such that it isn’t touched anymore. Doing so will not pose very many challenges—pandoc has really great capabilities for walking its ast in order to facilitate exactly these kinds of changes. Unsurprisingly, the Walkable class resides over all things walkable; an abbreviated definition looks like this:

Specifically, as we’ll need to shell out to an external program, let us restrict our attention to the more general walkM function here. There is an instance

which will be all that we need. The necessary code now just materialises in front of our eyes:

Notice how a priori this would have type Pandoc -> IO Pandoc, but since we want to use it from Hakyll I’ve already inserted a call to unsafeCompiler in the correct place.

Further, the above code checks whether the block has an explicit language attached to it and, if not, leaves it alone; this was suggested by LSLeary on Reddit. If you want to have a single div class for every code block—say, for some custom css—then you can replace

with

Hakyll¶

Thankfully, integrating pygmentsHighlight into Hakyll is not very complicated either. In addition to the normal pandocCompiler or pandocCompilerWith functions that you are probably already using, there is also pandocCompilerWithTransformM:

Basically, in additions to reader and writer options, it also takes a monadic transformation of pandoc’s ast and builds an appropriate Compiler from that.

The myPandocCompiler function can now be used as any other compiler; for example:

For a full working example, see my configuration.

Conclusion¶

That’s it! To my eyes, syntax highlighting looks much better now, and on the way I—and perhaps you as well—even learned a little bit about how pandoc internally represents its documents. Time well spent. As I said in the beginning, in principle one could swap out pygmentize for any other syntax highlighter that can produce html. However, for me these results are good enough that I will probably not try out every tool under the sun, chasing that ever present epsilon of highlighting cases which I still don’t agree with—at least for now.