From xWiki to static-HTML. How we “transferred” documentation

HOSTKEY documentation consists of two parts: internal, which is maintained and supplemented both by our technical writers and by the staff of the departments for which it is intended, and external, client-side.

Until recently, we used xWiki for both internal and external documentation. And while its use for internal documentation is justified, for external documentation xWiki is not the most optimal solution: external documentation is created by two or three people at most, registration of additional users is not necessary, xWiki is constantly under attack from spam bots, and changing the structure of documentation, renaming, mass-editing, changing the design and some other things require either edits directly in the database, or quite a lot of body movements. The pluses are a visual editor, the ability to import/export articles in OpenDocument format and flexible setting of permissions. Therefore it was decided to move to a new engine for external client documentation, and the choice fell on Material for MkDocs.

The whole move took us two months, from working out the structure and refactoring to the migration and translation of some articles.

Why Material for MkDocs?

Material for MkDocs is a powerful documentation creation tool that offers many technical features. It is based on Mkdocs, a popular tool for creating documentation, and Material Design, a modern and elegant design style from Google.

What attracted us to it?

1. All documentation is text files and can be easily stored in GIT, so it is possible to create branches for new sections that are still being worked on, to be able to quickly roll back documentation to a certain commit, and other advantages of using a distributed version control system. But you can just work with the documentation catalog and article files in it without GIT.
2. Collaborative and localized work. People working on documentation can use the same repository and not interfere with each other. It is also possible to work on documentation without connecting to the network at all and then only copy and upload it to the repository.
3. Static-HTML output. The documentation is assembled into a static html file structure (with inbuilt search) and is simply "put" into a specific section of the site.
4. Possibility of customisation. Quite a large number of display settings, including settings for individual pages, and the ability to easily customize the appearance through css. If you want more, you can completely rebuild the theme.
5. Markdown markup with the ability to include HTML code. If necessary, you can correct the text in any text editor, and also include missing elements in the documentation that cannot be done by MkDocs plugins. For example, iframe for inserting YouTube videos or beautiful page design.
6. Adaptive layout of documentation for both mobile devices and desktop.
7. Convenience of working on multilingual documentation and high speed of its changes. Since all articles are text files, changing the name of a parameter in the hosting control panel in the case of xWiki would force either to change it in the database or to search all articles for its occurrence. In the case of Material for MkDocs it is enough to search and replace in the file structure.
8. Permanent structure. Documentation at the physical level is a structure of files and catalogs. Therefore, when changing the navigation, section names, changing their order, either nothing needs to be changed in the file structure, or it is enough to copy the files to another catalog and replace the paths in the links by search-replacement on the files.
9. It is convenient to maintain multilingual documentation using a similar file structure. Add an article in one language, create and translate the same file in another language structure.
10. Built-in full-text search of all articles with the output of matches from the most relevant ones.

There are cons as well, and we were prepared for them (and they're something to consider if you want to switch to Material for MkDocs too):

1. You work with text files. That is, you write text and mark it up with special character sets and you don't see the final result immediately. Yes, there are editors with on-the-fly Markdown rendering, but since Material for MkDocs has a different part of the markup, you don't see the final result as you type. This is something you have to get used to and have an idea of what you want to get.
2. You need to know markdown markup with contrivances from the engine authors.
3. Images should be copied into the file structure of your documentation and links to them should be inserted into the text.
4. When using GIT, you need to know how to work with it.
5. Since the engine is written in Python, the entire layout is tied to tabs. Sections, indentations, numbering, etc. require you to be able to keep track of tab levels. Navigation in a yml file does too.
6. The user manual is in English. For example, there have already been many articles on its installation on "Habra", but you will have to look for some points in English (and sometimes even in the Issue-section of the project repository).
7. It is highly desirable to work on documentation locally. And this will require you to install Python, pip, mkdocs packages, GIT and an editor for easy work.
8. The engine builds documentation in its entirety, even when it is pre-rendered for viewing. If you have a large amount of documentation, you will have to wait for ten or twenty seconds for the reassembly of the preview when you make a couple of commas. And you will see the final version only when you build the documentation on your server, because the appearance will be affected by the general CSS-style of the site.
9. If you need to change the layout, you need to rebuild the theme.
10. Some of the interesting features are available only in the paid version. Although they can be reproduced by inserting html code into article files.
11. When creating links to certain headings (sections) within a page, only English language is taken into account. For example, the English link to the title Instant Servers of the article in the index.md file will be /apidocs/index.md#instant-servers

How do we do things?

Our documentation is available in multiple languages. You can access different language repositories to view the content. All languages are stored in a common repository, but each language has its own file branch. This allows for easy addition of support for new languages in the future by copying the language subfolder, renaming it, and translating the articles. The repository has two branches: dev and master. The dev branch is where most of the writing work is done and documentation for stage servers is collected. The master branch is where changes from the dev branch are merged. The repository itself is stored on its own GitLab.

Technical writers work with documentation locally (for them there is an instruction on how to deploy Material for MkDocs on a local Windows and Linux machine and work with it) and use Visual Studio Code (or VSCodium) for this purpose, fluffing commits only after making changes and checking everything on their machines. This makes it convenient to work with markdown markup and the GIT repository without having to use a terminal.

To run the pre-rendering, bat/sh files have been created that run it on localhost for ru- and en- vectors on different ports, which allows you to open and see the result for both languages at the same time. In the terminal for running these scripts you can also see warnings and errors for navigation problems or missing links to files and images.

The main rendering is done by the Jenkins task, where the path and rendering language (stage or prod) are selected and the GIT repository is downloaded, the required language directory is built and placed on the site. Placing our amount of documentation on the site takes about two minutes, direct building by Material for MkDocs itself takes about forty to fifty seconds.

Useful points

Since a lot of articles are devoted to installing Material for MkDocs, including those on "Habr" (there is a docker-image, there are ready pip-packages, which allows you to install it under any OS and there is a good instruction from the developers), I would like to tell you about the little things, some of which are obviously not described:

1. Useful plug-ins for Visual Studio Code. In principle, when working on documentation, you can do without them. For example, the plugin for checking yml-file configuration and navigation recommended by the authors of Material for MkDocs doesn't give a great advantage in speed and quality of work, and in practice it is more of a hindrance. The same can be said about various plugins for Markdown markup. But if you want to get competent documentation, we recommend a spellchecker. An inbuilt translator will also be useful, so you don't have to switch to a browser (we have Google Translate, but look at paying for the Deepl API and using its plugin).

   

2. Work out the design code for your documentation in advance. For example, we have such code. It is impossible to remember all the markup (especially the names of icons and icons), and this document will be useful for newcomers. We have it duplicated in the README of the git repository, so you can open this file and copy the necessary part of the markup.

   

3. You can "tweak" the appearance of your documentation through the /stylesheet/extra.css file. This came in handy for us when Material for MkDocs didn't have markup for buttons (only for keyboard keys), so we added this code there and use strikethrough (~~**button_name**~~) instead of buttons.:

   /* apply del element instead of button (~~text~~) */
   del {
   	text-decoration: none;
   	background-color: #FFA500;
   	color: #000;
   	padding: 2px 4px;
   	border: solid #343a40 2px;
   }

   In the same way, without rebuilding the theme, we matched the documentation design to the style of the site, expanded its size and made other minor adjustments.

4. Undocumented features include calling headers of a certain type and level from the nav section of the mkdocs.yml configuration file. We applied this in the case when we needed to refer in the navigation to the Virtual Servers section, which is located in the /apidocs/index.md file. To do this, in the nav-file it is enough to place the following text:

    - Virtual servers: apidocs/#virtual-servers

   Yes, when building, the system will swear that it cannot find this path, but in fact everything will work, because index.md will be associated with the apidocs section in the menu, and the section we need will be found in it.

5. If you need to create a link to a certain section, then according to the documentation you should create an index.md file in it, but then when you enter such a section you will have a picture like this:

   

   There are two options here: either write a disclaimer so that people reading your documentation don't get lost and pay attention to the left menu (or click the button that opens it on a mobile device) or redirect to the first article in the section.

   For the latter, you just need to place this code in the index.md file:

   <head>
   <meta http-equiv="refresh" content="0;URL=./text1/" />
   </head>

   Where ./text1/ you replace with the partition file you want (without the .md extension).

6. Always place images in the image directory at the same level as the article that uses them and name them using the <image_image_filename.png> template. Then you will always have the markup for inserting images will have the form , and if you need to move the article to another section, it will be enough to copy the md-file of the article and move images from one image directory to another, without changing anything in the markup itself.

7. Don't be afraid to use iframes in the text of md-files of articles. The engine digests them perfectly well. So, on this page the video is embedded with an iframe.

   

8. Use tags. According to some peculiarities of the engine, pages with tags are given priority in search. Tags can be added through such a construction at the beginning of the article file:

   ---
   tags:
   - Business
   - Software Marketplace
   ---

9. To control the structure of section headings, pay attention in VS to the Outline panel, which displays the entire article structure perfectly.

   

10. For changes, you can use the following algorithm, which we follow:

    • We make changes and edits to the dev branch during the day, collect them for testing on stage, agree on them, and at the end of the day (in the evening), if they are available, merge the dev and master branches and collect them on the site.
    • Urgent changes and edits are made in the master branch with a mandatory backport to the dev branch. We apply such changes only if without these changes the documentation may lead to serious problems.
    • If new articles or sections are being prepared that cannot be put into the documentation yet, we create separate branches from dev, work in them, then merge them back and delete this branch. We show and coordinate such documentation either through local build and sharing the result via Google Meet or building a branch on stage.

You can see what we've done here.

And what is your documentation "built" on?