sysman Configuration¶

Introduction¶

The sysman is a collection of manuals for all users in the MI and CI group. Each manual is constructed with Markdown and presented with MkDocs as a website.

This manual explains how the sysman is configured.

If you have an account on the gitlab-server of the campus you can access the git-repository with all source files of sysman:

• https://gitlab.tudelft.nl/rligteringen/sysman

Workflow¶

All manuals are written in plain markdown combined with extra functionality provided by mkdocs extensions and plugins. Writing of the manuals is done with code (Visual Studio Code) using the markdownlint and Markdown Preview Enhanced extensions. Also the mkdocs configuration files are edited in code.

After adding or changing the source files the mkdocs application is used to build the website. Inside the docs directory all .md- and .ipynb-files are rendered as .html-files inside the site directory together with the MkDocs website environment. The application allows you to test your website or to build it for distribution.

All data is stored in a git-repository on https://gitlab.tudelft.nl. After creation of the website with mkdocs it is published on https://qiweb.tudelft.nl/sysman. This process is automated in GitLab using CI/CD. The configuration file for this continues integration/deployment can be found in the git-repository with the name .gitlab-ci.yml. For this a gitlab-runner is installed on the server gitlab-imphys.tudelft.nl. More information can be found in the chapter below about Continues Deployment.

Writing can be done everywhere as long as git is available for pulling and pushing the data. Testing can also be done everywhere provided a python-environment is set up with the mkdocs-package, see below for more information.

Setting up MkDocs environment¶

# obtain copy of repository
$ git clone git@gitlab.tudelft.nl:rligteringen/sysman.git
$ cd sysman
# setup virtual environment in Conda with mkdocs package
$ conda create -n sysman pip
$ conda activate sysman
$ pip install -r requirements.txt

Using material-theme¶

This theme has a very nice layout. More information:

https://squidfunk.github.io/mkdocs-material/

Add to mkdocs.yml:

theme:
  name: 'material'

Plugins¶

git-revision-date-localized¶

Add to mkdocs.yml:

plugins:
  - search
  - git-revision-date-localized

Extensions¶

Please check the mkdocs.yml for the extensions that have been added to mkdocs. Also note the use of mathjax for rendering \(\LaTeX\) equations.

Testing mkdocs output¶

Before publishing your website you can test it on your local computer using a live build together with a small webserver. For this you have to give the following command inside the mkdocs directory sysman:

$ mkdocs serve

Now open a webbrowser to http://127.0.0.1:8000 and look at the mkdocs-site. If you keep this program running all changes you make to the source markdown files will be automatically shown in the browser.

Continues Deployment¶

After each push to the gitlab-repository an automated process is executed using the configuration file .gitlab-ci.yml:

stages:
  - build
  - deploy

variables:
  ### needed for mkdocs plugin: git-revision-date-localized
  GIT_DEPTH: 1000

build_mkdocs:
  # image: python:alpine
  image: ronligt/mkdocs:3
  stage: build
  script:
  - mkdocs build
  artifacts:
    paths:
      - site
    expire_in: 1 week
  only:
  - master

deploy:
  # image: alpine
  image: ronligt/mkdocs:3
  stage: deploy
  script:
  - eval $(ssh-agent -s)
  ## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
  ## We're using tr to fix line endings which makes ed25519 keys work
  ## without extra base64 encoding.
  ## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
  ##
  - echo "$KEY" | tr -d '\r' | ssh-add - > /dev/null
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - ssh-keyscan qiweb.tudelft.nl >> ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts
  - rsync -rv --delete site/ web@qiweb.tudelft.nl:sysman/
  only:
  - master

Runners¶

The two different stages are executed sequentially by the runner(s) configured for this gitlab project. The runners run on separate servers and ‘announce’ themselves to the project on the gitlab-server during registration. For this the hostname of the gitlab-server and the token for the project is needed. These can be found in gitlab:

The runner is installed on gitlab-imphys.tudelft.nl with the following commands:

$ sudo wget "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb"
$ sudo dpkg -i gitlab-runner-amd64.deb

See also https://docs.gitlab.com/runner/install/linux-manually.html

And registered to the gitlab server with the following commands:

$ sudo gitlab-runner register
## Follow all instructions and fill in the hostname and token found in the project
$ sudo gitlab-runner install
$ sudo gitlab-runner start

See also https://docs.gitlab.com/runner/register/

In the deployment phase an rsync is used to copy the site build with mkdocs to the webserver qiweb.tudelft.nl. This rsync is done over SSH and uses a private-public key authentication. This key is created on the webserver. The private key (!) is stored in gitlab in the variable KEY. This variable is then used by the runner to authenticate and synchronize the website.

Docker container¶

The runner uses a docker container which can be created with the Dockerfile in the root of the repository:

# Dockerfile for creating sysman-pages in gitlab-runner
FROM python:alpine

RUN apk update --no-cache
RUN apk add --no-cache git openssh-client rsync
RUN apk add --no-cache gcc g++

RUN pip install --upgrade pip
COPY requirements.txt /
RUN pip install --no-cache-dir -r /requirements.txt

RUN apk del --no-cache gcc g++

You can update this container as follows:

$ cd sysman
$ docker build -t ronligt/mkdocs:<version> .
$ docker push ronligt/mkdocs:<version>

Make sure change the version number in the .gitlab-ci.yml file! Also clean up the previous container on the runner server:

# show all images
$ docker images
$ docker rmi ronligt/mkdocs:<previous_version>

Formatting¶

keyboard entries (HTML5)

<kbd>shift</kbd> gives shift

keyboard entries (pymdownx.keys)

++ctrl+c++ gives Ctrl+C

arrows (pymdownx.smartsymbols)

--> gives →, check https://facelessuser.github.io/pymdown-extensions/extensions/smartsymbols/

emoji (pymdownx.emoji)

:smile: gives , check https://www.webfx.com/tools/emoji-cheat-sheet/

highlight code (pymdownx.highlite with pymdownx.superfances)

hl_lines="1 3" highlights line 1 and 3 in the following example:

```python hl_lines="1 3"
x = np.linspace(0,N-1,N)
y = noise
std = np.zeros(N)
```

gives:

x = np.linspace(0,N-1,N)
y = noise
std = np.zeros(N)

diagram (mermaid2)

check out https://mermaid-js.github.io/mermaid/#/, see the following example:

```mermaid
graph TD;
A-->B;
A-->C;
B-->D;
C-->D;
```

gives

graph TD; A-->B; A-->C; B-->D; C-->D;