IBM Unveils DocLang: A New Markup Language for Large Language Models

IBM Research has released DocLang, an open-source markup language designed specifically for Large Language Models (LLMs). This new standard is intended to improve machine understanding of documents and reduce latency and inferencing costs.

DocLang builds on the success of IBM's previous project, Docling, a document parser with 32 million downloads and 64,000 stars on GitHub. The researchers behind Docling observed that existing document formats were not designed for machine understanding and often required multiple passes to extract relevant information.

DocLang addresses this issue by providing a clean semantic-token link, where each token has an explicit opening and closing form. This allows LLMs to directly interpret graphics and other visual elements without the need for additional processing.

The benefits of DocLang include higher accuracy with smaller models, lossless and grounded representation, consistency, and governance built-in. The standard is also open, with no vendor lock-in, making it accessible to a wide range of users.