Quick Start KVM libvirt VMs with Terraform and Ansible – Part 2

In the first part of my Quickstart Guide, I showed how to use Terraform Script to start new VMs or guests under KVM libvirt very flexibly and quickly.

Once the virtual machines are up and running, Ansible can be used to configure them and install additional software. Ansible is an open-source configuration management and administration tool to automate recurring work on systems. It is virtually the admin’s best friend to relieve him of tedious maintenance work. Ansible is an easy to learn and read script description language in YAML format. It requires very few resources and is very flexible. Ansible works agentless. This means that no Ansible agent needs to be installed on the target systems (as is the case for Puppet, for example). Instead, Ansible uses the SSH connection with private/public keys for secure communication with the target systems on Linux systems, for example.

Ansible uses so-called playbooks (YAML format and syntax) in which the plays are described as tasks that are to be processed on the target systems. In addition, there is an inventory file in which, among other things, the host names are stored. Recurring tasks can also be outsourced to modules. Ansible already offers a lot here itself to handle the standard tasks, but you can also develop your own modules. Ansible playbooks are also idempotent. That is, a playbook can be executed multiple times and always returns the same result. A playbook thus describes the target state of a system.

But let’s finally get started …

Preparations and installation

Installing Ansible on Linux is easy, as you can also see here: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-ansible-on-rhel-centos-or-fedora


juergen  ~  sudo dnf install ansible

Letzte Prüfung auf abgelaufene Metadaten: vor 0:37:27 am Mi 18 Nov 2020 15:23:03 CET.
Abhängigkeiten sind aufgelöst.
======================================================================================================================================================
 Package                                      Architecture                  Version                              Repository                      Size
======================================================================================================================================================
Installieren:
 ansible                                      noarch                        2.9.14-1.fc32                        updates                         15 M
Abhängigkeiten werden installiert:
 libsodium                                    x86_64                        1.0.18-3.fc32                        fedora                         169 k
 python3-babel                                noarch                        2.8.0-2.fc32                         fedora                         5.7 M
 python3-bcrypt                               x86_64                        3.1.7-4.fc32                         fedora                          44 k
 python3-fluidity-sm                          noarch                        0.2.0-18.fc32                        fedora                          19 k
 python3-jinja2                               noarch                        2.11.2-1.fc32                        updates                        490 k
 python3-jmespath                             noarch                        0.9.4-4.fc32                         fedora                          46 k
 python3-lexicon                              noarch                        1.0.0-10.fc32                        fedora                          18 k
 python3-ntlm-auth                            noarch                        1.1.0-9.fc32                         fedora                          50 k
 python3-pynacl                               x86_64                        1.3.0-6.fc32                         fedora                         102 k
 python3-pyyaml                               x86_64                        5.3.1-1.fc32                         fedora                         202 k
 python3-requests_ntlm                        noarch                        1.1.0-10.fc32                        fedora                          18 k
 python3-xmltodict                            noarch                        0.12.0-7.fc32                        fedora                          23 k
Schwache Abhängigkeiten werden installiert:
 python3-invoke                               noarch                        1.4.1-1.fc32                         fedora                         149 k
 python3-paramiko                             noarch                        2.7.1-2.fc32                         fedora                         287 k
 python3-pyasn1                               noarch                        0.4.8-1.fc32                         fedora                         133 k
 python3-winrm                                noarch                        0.3.0-9.fc32                         fedora                          59 k

Transaktionsübersicht
======================================================================================================================================================
Installieren  17 Pakete

Gesamte Downloadgröße: 23 M
Installationsgröße: 131 M

With ansible –help you can check if the installation was successful.

Ansible requires an SSH connection to communicate with the target systems under Linux, which is established via SSH keypair. To do this, we first create our own user.

 juergen  ~  sudo useradd -m ansible

 juergen  ~  sudo passwd ansible
ändere Passwort für Benutzer ansible.
Geben Sie ein neues Passwort ein: 
Geben Sie das neue Passwort erneut ein: 
passwd: alle Authentifizierungsmerkmale erfolgreich aktualisiert.

Then we create an SSH keypair for the Ansible user.

 juergen  ~  su - ansible
Passwort: 
[ansible@localhost ~]$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ansible/.ssh/id_rsa): 
Created directory '/home/ansible/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ansible/.ssh/id_rsa
Your public key has been saved in /home/ansible/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:2          ansible@localhost.localdomain
The key's randomart image is:
+---[RSA 3072]----+
|      .          |
|      o          |
+----[SHA256]-----+
[ansible@localhost ~]$ ll .ssh
insgesamt 8
-rw-------. 1 ansible ansible 2622 18. Nov 16:13 id_rsa
-rw-r--r--. 1 ansible ansible  583 18. Nov 16:13 id_rsa.pub
[ansible@localhost ~]$ 

If we set up our VMs via Terraform, as described in the first part of my installation instructions, it makes sense to provide the public key immediately. We copy the content of the id_rsa.pub file into the Terraform cloudinit.cfg file, as described in part 1, and thus distribute the public key to all new target systems. You can also do this manually at any time later. To do this, however, you must first create an ansible user on each system and then distribute the public key to all target servers from the ansible server as the ansible user using the ssh-copy-id command.

Ansible project files

First we need a project folder, in my example “newsrv”, where we will create the Ansible files. I will explain the required files in detail below and would also like to encourage you to make your own extensions.

ansible.cfg

Then we create a configuration file named ansible.cfg for general Ansible settings of our project. Besides that, you can also make your settings in the global configuration file under /etc/ansible, which then applies to all projects equally. In my example config, it is worth mentioning that I set host_key_checking=false. This reduces the security of the SSH connection, but in my case it is helpful if you recreate the VM often, because then the hostkey logically changes every time.

[defaults]
hash_behaviour=merge
host_key_checking=false

[ssh_connection]
pipelining = True
timeout=30

Then we have to copy the private SSH key of the ansible user we created above into the project folder. This is how the executing ansible user authenticates himself to the target systems.

 juergen  ~  Scripts  ansible  newsrv  sudo cp /home/ansible/.ssh/id_rsa .

 juergen  ~  Scripts  ansible  newsrv  sudo chown juergen:juergen id_rsa

hosts.yml

Next, we create a hosts.yml file that contains the hostname(s) of the target servers and some global variables.

all:
  vars:
    ansible_connection: ssh
    ansible_user: ansible
    ansible_become: true
    ansible_ssh_private_key_file: id_rsa
    ansible_python_interpreter: /usr/bin/python3

  hosts:
    mysrv.mydomain.vm:

install.yml

Then we need a playbook that executes the actual tasks. I will explain the file here again step by step. My playbook contains on one side settings and configurations that I basically want to do for all VMs. And then there are also some examples for software installations in the playbook. This should encourage you to create your own tasks and playbooks. Because if you have understood the principle, it is not difficult to extend the playbook according to your ideas.

Note: A yaml file is always prefixed with three hyphens — and strict attention must also be paid to the correct indentations.

First, the playbook gets a name, in my example not very creative: “Install some packages”. hosts: all indicates that the playbook should be applied to all defined systems and tasks: finally introduces the following task blocks.

---
- name: Install some packages
  hosts: all
  tasks:

In the first task block, IPv6 is completely disabled as it is redundant in my VM environment.

# Disable ipv6
   - name: Disable IPv6 with sysctl
     sysctl: name={{ item }} value=1 state=present reload=yes
     with_items:
       - net.ipv6.conf.all.disable_ipv6
       - net.ipv6.conf.default.disable_ipv6
       - net.ipv6.conf.lo.disable_ipv6
   
   - name: RedHat | placeholder true for ipv6 in modprobe
     lineinfile: "dest=/etc/modprobe.conf line='install ipv6 /bin/true' create=yes"
     when: ansible_os_family == 'RedHat'
   
   - name: RedHat | disable ipv6 in sysconfig/network
     lineinfile:
       dest: /etc/sysconfig/network
       regexp: "^{{ item.regexp }}"
       line: "{{ item.line }}"
       backup: yes
       create: yes
     with_items:
       - { regexp: 'NETWORKING_IPV6=.*', line: 'NETWORKING_IPV6=NO' }
       - { regexp: 'IPV6INIT=.*', line: 'IPV6INIT=no' }
     notify:
       - restart NetworkManager.service
     when: ansible_os_family == 'RedHat'

Then all packages are brought up to date. If you want to serve different Linux derivatives with the same playbook, then you have to include a query of the OS family, because the operating system families have known different package managers (Debian: apt, SuSe: zypper …).

If you also include so-called “tags” in your playbook, then you can later execute only certain parts of your playbook specifically. This is useful for the upgrade task, for example, because this process is performed regularly.

The debug task outputs the content of the variable “result.changed” to the console. This is helpful if you don’t know the passing value from the beginning and thus want to output it. Further down in the playbook, the status of this variable then becomes relevant for the decision whether to reboot the server at the end.

# Upgrade all packages
   - name: upgrade all packages on CentOS/Fedora
     yum:
       name: "*"
       state: latest
     when: ansible_os_family == 'RedHat'
     register: result
     tags:
       - upgrade
   - debug:
       var: result.changed
     tags:
       - upgrade

Then some packages I would like to have on all my systems are installed and the correct timezone is set.

# Some usefull packages for CentOS
   - name: install bind-utils for CentOS
     package:
       name: bind-utils
       state: present
     when: ansible_os_family == 'RedHat'
   - name: install cronie to create cronjobs for CentOS
     package:
       name: cronie
       state: present
     when: ansible_os_family == 'RedHat'


# Install for all distributions Debian/Redhat
   - name: install cockpit to monitor server
     package:
       name: cockpit
       state: present
   - name: install bash-completion 
     package:
       name: bash-completion
       state: present

# Set timezone
   - name: set timezone
     timezone:
       name: Europe/Berlin

As an example application the CentOS software group “Server with GUI” and Firefox as browser is installed here.

# Install Server with GUI
   - name: install server with GUI for CentOS
     package:
       name: '@graphical-server-environment'
       state: present
     when: ansible_os_family == 'RedHat'

# Install Webbrowser
   - name: install firefox
     package:
       name: firefox
       state: present
     when: ansible_os_family == 'RedHat'

It would also be nice if autologin was enabled for the server. Finally, a reboot is performed if the upgrade task variable returns true.

# Activate autologin to console for a user
   - name: Creates getty tty1_service_de directory
     file:
       path: "/etc/systemd/system/getty@tty1.service.d"
       state: directory
       owner: root
       group: root
       mode: 0775
     tags: 
       - autologin
   - name: Activate console autologin for a user
     copy:
       dest: "/etc/systemd/system/getty@tty1.service.d/override.conf"
       owner: root
       group: root
       mode: 0644
       content: |
         # Provided by Ansible
         [Service]
         ExecStart=
         ExecStart=-/sbin/agetty --noissue --autologin juergen %I $TERM
         Type=idle
       backup: yes
     tags: 
       - autologin



# Reboot when all things done
   - reboot:
       msg: "Reboot to finish setup."
       reboot_timeout: 60
     when: result.changed == "true"

Some important Ansible commands

Now that we have all the Ansible files together in our project folder, we can use the following command to display all the hosts in this project, for example:

 juergen  ~  Scripts  ansible  newsrv  ansible --list-hosts all -i hosts.yml
  hosts (1):
    mysrv.mydomain.vm

Next, we should test the connection to the target server(s) by doing an Ansible ping:

 juergen  ~  Scripts  ansible  newsrv  ansible -m ping all -i hosts.yml
mysrv.mydomain.vm | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

If that was successful, then it’s time to run our playbook.

 juergen  ~  Scripts  ansible  newsrv  ansible-playbook -i hosts.yml install.yml

If everything doesn’t work right from the start, don’t give up right away. Often it is very banal errors that prevent the execution. For example, in the beginning it often happened to me that I made mistakes with the indentations in the YAML code, because I copied the code directly from the Internet. At the beginning it is also possible that the SSH connection setup via key-pair does not work correctly. Most of the time googling helps.

This is what the Ansibe output looks like during the Playbook run.

The tags in the playbook can also be used to execute only parts of the playbook, such as the upgrade task. This allows you to perform specific tasks without always having to run the entire playbook.

 juergen  ~  Scripts  ansible  newsrv  ansible-playbook -i hosts.yml install.yml --tags upgrade

PLAY [Install some packages] ***************************************************************************************

TASK [Gathering Facts] *********************************************************************************************
ok: [mysrv.mydomain.vm]

TASK [upgrade all packages on CentOS/Fedora] ***********************************************************************
ok: [mysrv.mydomain.vm]

TASK [debug] *******************************************************************************************************
ok: [mysrv.mydomain.vm] => {
    "result.changed": false
}

PLAY RECAP *********************************************************************************************************
mysrv.mydomain.vm          : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

This brings us to the end of the second part. I hope I could inspire you to manage your KVM/libvirt infrastructure via Terraform and Ansible in the future. I would be happy if you give me your feedback.

Read also the first part of this Quick Start guide.