Python 3 for System Administrators: The Plot!

📓 Shell or Python? Should I replace my Shell scripts with Python?

It’s debatable! 

That’s what Nick, who has been a mentor for Ajay, added.

It certainly depends, whether you work in Sysadmin, Cloud operations, DevOps or SRE. But that love for Shell would always be there. We all started with it and it’s the glue!

Ajay: But isn’t Shell scripts slow? In addition, it lacks data structure and gets highly complex with long scripts.

Nick: Maybe, but that’s not the point 😄 There are day-to-day tasks that can be performed in both, say checking application ports, starting/stopping the server, checking the resources consumption/load, backing up your database and exporting it to object storage.

🐍 Python is clean, it feels like writing pseudo code. In other words it’s easier to write and maintain, provides better error handling, can perform OOP (object-oriented programming), and the best part is argument parsing and modules.

🐚 Shell is something that we all have started with, hence I describe it with words “omnipresence” or “has ubiquity”. Quick filtering or command chaining with Pipe (”|”) and CLI utilities like sed, awk, grep, etc.

Ajay: Awesome, let me give it a shot; it would certainly help adding new skills to my knowledge bank, at the very least.

Hence, Ajay- who has been a sysadmin all the while writing scripts in shell, just wants to try Python and see how he can make use of both to add more efficiency to his day-to-day work.

So, this is Ajay’s review on linux academy’s course for Python which he found out while browsing the internet and it simply said: “Learn or start only with modules and structure that would be useful for you as Sysadmin and not the whole Python at once!”

Ajay can be any person who has spent a good time with shell, and not specially sysadmin. So this is for anyone who just wants to add an element to fun to work and has been wanting to learn Python for a long time!

💡 Linux academy is an online education platform and has some cool stuff, if you are creating or analyzing solutions in the Cloud. A lot about them can be read here(https://linuxacademy.com/about/).

Note: 
This is in no way a connection or recommendation from me to try 
only Linux Academy; I just loved this course and this is an 
honest review. 

The following is an introduction to the course, where I’ll describe each section, add my inputs for code conversion, using specific modules and the outcomes!

Course: Python 3 for System administrators

Author: Keith Thompson

Course Introduction:

Getting started | Environment setup | Introducing Python

Keith has done a fantastic job of making sure your base is set up strong here. That means, setting up a cloud development server and taking care of small small things, like having a custom setup for ~/.bashrc and ~/.vimrc.

Customizing bash prompt:
$ curl https://raw.githubusercontent.com/linuxacademy/content-python3-sysadmin/master/helpers/bashrc -o ~/.bashrc

Customizing Vim:
$ curl https://raw.githubusercontent.com/linuxacademy/content-python3-sysadmin/master/helpers/vimrc -o ~/.vimrc

Now, these are completely optional you can have your own setup and can go with more advanced stuff like the ultimate Vim configuration: https://github.com/amix/vimrc

Setting up and customizing stuff- I feel, sets the tone right away that Keith wants you to have fun customizing stuff and getting in the same tone as himself. This section ends with history & benefits of python along with the most important thing, that is the comparison between Python 2 and Python 3.

Just Enough Python:

Running Python | Common Data Types | Control Flow

Now, you won’t jump into writing scripts right away, so the first step is to understand how python interprets stuff. REPL (Python Interactive Shell) is used here for rapid experimentation.

Common data types like Strings, numbers, Booleans, working with variables, lists, tuples, dictionaries.

Control Flow, mostly talks about loops, comparisons and logic operations. If you are like me who had started to learn python a lot many times and failed to move after the basic step this would be something you have already done. But, I would suggest to quickly go through this, as it could be a refresher.

Scripting with Python 3: the fun part starts here:

Scripting with Python 3 is divided into 2 different parts and that’s 
the beauty of the course.
Basic Scripting | Intermediate Scripting

Basic scripting

It’s more about reading user inputs, functions, interacting with files and working with environment variables. Obviously all of these are important, although the super helpful part to me as a sysadmin was the topic on interaction with files.

Fun exercises in course:
Write a script that does the following:
1. Takes a port_number as its only argument.
2. Calls out to lsof to determine if there is a process listening 
on that port.
3. If there is a process, kill the process and inform the user.
4. If there is no process, print that there was no process running 
on that port.

Intermediate Scripting:

This is where you can start to see the heat of chapters to come, I mean they are not very complicated but it needs a lot of attention and practice.

Parsing command line parameters, Handling errors, Exit status, Execute shell commands with Python!

Yeah, that sounds familiar and interesting right? When I first saw the course structure, I was eagerly waiting for the part that said “Execute shell commands with Python”. It was fun learning the “subprocess” module.

The subprocess module

The subprocess.run function

The bytes type

>>> import subprocess
>>> proc = subprocess.run(['ls', '-l’])

The proc variable is a CompletedProcess object, and this provides us with a lot of flexibility. We have access to the returncode attribute on our proc variable to ensure that it succeeded and returned a 0 to us.

>>> proc = subprocess.run(
...     ['ls', '-l'],
...     stdout=subprocess.PIPE,
...     stderr=subprocess.PIPE,
… )

>>> proc
CompletedProcess(args=['ls', '-l'], returncode=0, stdout=b'total 0\ndrwxrwxr-x. 2 sachcode sachcode   6 Mar 12 13:53 bin\ndrwxrwxr-x. 3 sachcode sachcode  22 Apr 11 13:50 code\ndrwxrwxr-x. 5 sachcode sachcode 170 Apr 22 18:17 hr\ndrwxrwxr-x. 2 sachcode sachcode  50 Apr 11 12:39 install-scripts\ndrwxrwxr-x. 3 sachcode sachcode 223 Apr 10 19:32 sample\n', stderr=b'')

>>> proc.stdout
b'total 0\ndrwxrwxr-x. 2 sachcode sachcode   6 Mar 12 13:53 bin\ndrwxrwxr-x. 3 sachcode sachcode  22 Apr 11 13:50 code\ndrwxrwxr-x. 5 sachcode sachcode 170 Apr 22 18:17 hr\ndrwxrwxr-x. 2 sachcode sachcode  50 Apr 11 12:39 install-scripts\ndrwxrwxr-x. 3 sachcode sachcode 223 Apr 10 19:32 sample\n'

>>> print(proc.stdout.decode())
total 0
drwxrwxr-x. 2 sachcode sachcode   6 Mar 12 13:53 bin
drwxrwxr-x. 3 sachcode sachcode  22 Apr 11 13:50 code
drwxrwxr-x. 5 sachcode sachcode 170 Apr 22 18:17 hr
drwxrwxr-x. 2 sachcode sachcode  50 Apr 11 12:39 install-scripts
drwxrwxr-x. 3 sachcode sachcode 223 Apr 10 19:32 sample

Now, that was a long talk about the subprocess module. I did that because I took a break from this course for 2 days and could see that subprocess was the bread and butter for us- the sys admins. There are some fun exercises in this section, so you’ll have fun please don’t skip them.

Useful standard library packages and Third party packages:

Standard: random & json | shutil & glob | re & math
Third party: Using Pip & Virtualenv

pip is the reference Python package manager and virtualenv is a tool that creates an isolated environment in Python.

$ pip install virtualenv
$ pip list
$ mkdir sample
$ cd sample
$ virtualenv sample_project
$ source sample_project/bin/activate

Docs:

https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/

https://docs.python-guide.org/dev/virtualenvs/

The practical exercise(s) in this section is not something that you would want to skip, as they are fun!

🚀 It would be stuff like taking a simple URL in the script and parsing the output based on what anyone wants, say JSON or HTML.

Creating a larger scripting project:

Keith has been very frank in notifying that, the following section would be trickier and might need more attention; basically it won’t be easy stuff. And yeah it wasn’t, I had to do this part twice at least before I moved to the exercise to implement another project.

Project:
We have many database servers that we manage, and we want to create a 
single tool that we can use to easily back up the databases to either 
AWS S3 or locally. We would like to be able to:

> Specify the database URL to backup.
> Specify a “driver” (local or s3)
> Specify the backup “destination”. This will be a file path for local 
and a bucket name for s3.
> Depending on the “driver”, create a local backup of the database or 
upload the backup to an S3 bucket.

:git: https://github.com/linuxacademy/content-python-for-sys-admins/tree/master/pgbackup

Usage:
..............
Pass in a full database URL, the storage driver, and destination.
S3 Example w/ bucket name:
$ pgbackup postgres://bob@example.com:5432/db_one --driver s3 backups

Local Example w/ local path:
$ pgbackup postgres://bob@example.com:5432/db_one --driver local \
/var/local/db_one/backups/dump.sql

It’s goes very deep in terms of setting up!

TDD (Test-Driven Development) and writing the first tests:

Test automation frameworks are famous for Python and majorly there are a couple of them that contribute to unit testing, like Robot, Pytest, Unittest or PyUnit. In this course Keith uses Pytest as the framework.

It really depends on what you would like to use and here’s a good comparison of some of them.

Techniques like stubs and mocks can be used to separate code into “units” and run unit/individual level testing.

📓 Stubs and Mocks: They both are dummy objects that simulate fake methods that are being tested.

A Stub is used to fill in some dependency required for unit test to run correctly. A Mock on the other hand is a fake object which runs the tests where we put assert.

$ pgbackup --driver local ./local-dump.sql \
postgres://demo:password@54.245.63.9:80/sample

$ pgbackup --driver s3 pyscripting-db-backups \
postgres://demo:password@54.245.63.9:80/sample

The pytest package

The pytest.raises function

We’re going to write three tests to start:
1. A test that shows that the CLI fails if no driver is specified.
2. A test that shows that the CLI fails if there is no 
destination value given.
3. A test that shows, given a driver and a destination, 
that the CLI’s returned Namespace has the proper values set.

The course goes a but more deeper on the path of how we could implement the CLI guided by tests, how to write mocking tests, implementing PostgreSQL and local storage, and finally the AWS interaction.

Building and sharing the tool:

As the logic says we sysadmins won’t be sharing or open sourcing every little tool we write, but we will want it to be distributable. The newest and preferred way to distribute a python tool is to build a ‘wheel’.

Wheel documentation: link

Exercise part: another project:

The final part of the course has a set of exercises that is again a full project implementation. Just adding a bit of introduction on same, and didn’t wanted to be a full spoiler here:

1. Setting up the project’s directory structure and metadata.
2. Using TDD you will be writing tests, and building the CLI.
3. Implement the user management
4. JSON parsing and exporting
5. Working on the console part, as we did in the above guided project.
6. The final part of building a shareable project using Wheel.

This is the info with which Keith ends the course:

Now that you have a grasp of Python scripting, there are a lot of doors 
open to you to continue your learning. 

Dive into more configuration management using Ansible (which is written 
in Python), learn about data science, write more custom scripts, or 
maybe dive into web development.

That’s true, now this is not something in Python, but I’ve found a good library of sysadmin scripts that you can either use the same way or try changing them and practicing in Python.

📓 https://github.com/topics/sysadmin-scripts

Keith’s library for the course:

📓 https://github.com/linuxacademy/content-python-for-sys-admins

-------
Helpful resources/references:
-------

https://realpython.com/python-testing/

https://github.com/amix/vimrc

https://docs.pytest.org/en/latest/

https://dev.to/snird/the-difference-between-mocks-and-stubs-explained-with-js-kkc

https://stackoverflow.com/questions/2424921/python-vs-bash-in-which-kind-of-tasks-each-one-outruns-the-other-performance-w