da-2021-xproc

Presentation, exercises and notes for the XProc 3.0 tutorial at Declarative Amsterdam 2021

View the Project on GitHub xatapult/da-2021-xproc

XProc - A pipelining language - Declarative Amsterdam 2021

XProc 3.0 is an XML based programming language for complex document processing. Documents flow through pipelines in which steps perform processing like conversion, validation, split, merge, report, etc. It’s an almost perfect fit for the kind of complex processing necessary in XML document engineering.

Declarative Amsterdam 2021 (November 4 and 5, Amsterdam) will kick off with a day of tutorials and workshops. One of these will be an introduction to XProc 3.0.

The DA-2021-XProc GitHub repository is used to host all materials (presentations, exercises, etc.) for this introduction. It will be filled in the weeks leading up to the conference.

Contents

About the tutorial

The XProc 3.0 tutorial will be a mix of theory, demos and practice. If time and technical hurdles allow, attendees can do (simple) exercises to get practice and a feel for the language. The annotated presentation(s) will be available.

IMPORTANT: Preparations for the tutorial

If you would like to follow along and do some of the exercises yourself during the tutorial, you need a working XProc 3.0 processor present and working:

About XProc

What is XProc?

Let’s try to answer this question with an overview of XProc’s main high-level characteristics:

Now why and when would this be useful? In the physical world, pipelining and working in specialized steps is not unusual. Take for instance an oil refinery: it takes crude oil as its input and, through a series of steps and intermediate products, produces petrol/gasoline, kerosene, diesel, etc. Just one look shows that refineries take the word “pipeline” very literal…

A classic from the IT world are of course UNIX pipelines. Some command produces some output and we do further processing (by, for instance, grep or tail or head) to get the information needed. The character used for chaining steps, |, is even called the “pipe” character!

So what are applications of this in the world of information and document processing? One of the main reasons is that data is often not in the format we need it to be. Some examples:

For straight transformation of XML data there are languages available like XSLT and XQuery. But more often than not tasks are more complex than can be done in a single transformation: chaining, splitting an merging comes into play. Surrounding the transformations you need housekeeping, like where to read from or write to, inspect directories and zip files and write logs. Also from a software engineering point of view it is often desirable to work in steps to get more legible and better maintainable code. This is where XProc comes into play: a single executable language to express this.

Where to find more information?

Information about its predecessor XProc 1.0:

A little history

The previous version of XProc, XProc 1.0, has been around since 2010 but was never really widely used. People found it hard to comprehend, use and rather verbose. Learning materials were hard to find. However, those that climbed the learning curve used (and still use) it a lot. It’s an extremely useful tool for the complex kind of XML document processing that happens, for instance, in publishing.

In 2016 a W3C community group started working on XProc 3.0 (a 2.0 proposal with a non-XML syntax never got a lot of support and was therefore abandoned). Main goals were to stay closer to the 1.0 syntax, make the language much more usable, understandable and concise, update the underlying standards (most notably XPath) and allow processing of non-XML documents.

The XProc 3.0 specification is in “last call” status and very stable. We’re still working on some last minor details of some of the steps. There is one working processor (MorganaXProc-IIIse) and one on the way (the 3.0 version of XML Calabash).

We, the people involved in the XProc 3.0 initiative, believe that we’ve turned an old-fashioned, hard to use contraption into an up-to-date and easy to use modern power tool. A tool that can replace many of the incomplete homegrown pipeline implementations that we know are out there.

About the presenter

My name is Erik Siegel. Xatapult is my (one-man) company, specialized in content engineering and XML processing. Most of my clients are in the publishing industry or involved in standardization.

I come from a technical IT background. Xatapult is deliberately looking for content and XML related projects on all levels: from the strategic use of standards to developing processing applications.

I do not just strive for the best solution from a business and technical perspective, but also for optimal understandability. Therefore, documentation is never an afterthought. I consider my ability to explain and clarify complex technical stuff, both live and in prose, as an important non-technical skill.

In my (recent) past I have given several successful courses on topics such as XML, XSLT and XML Schema, both in-house for clients and for commercial course providers. Most of these courses I developed myself.

In 2014, together with Adam Retter, I wrote a book about eXist-db, an XML database and processing engine. The book was published by O’Reilly and sold world-wide.

In 2016 I became a member of the editing committee for the XProc 3.0 standard. I am the author of the XProc 3.0 Programmer Reference, which was published in 2020.

I’m currently working on a book about Schematron, which is expected to be published in the second half of 2022.

More details on my LinkedIn profile.