April 25, 2020
Nick, who has been a mentor for
It certainly depends, whether you work in Sysadmin, Cloud operations, DevOps or SRE. But that love for Shell would always be there. We all started with it and it’s the glue!
Ajay: But isn’t Shell scripts slow? In addition, it lacks data structure and gets highly complex with long scripts.
Nick: Maybe, but that’s not the point 😄 There are day-to-day tasks that can be performed in both, say checking application ports, starting/stopping the server, checking the resources consumption/load, backing up your database and exporting it to object storage.
Python is clean, it feels like writing pseudo code. In other words it’s easier to write and maintain, provides better error handling, can perform OOP (object-oriented programming), and the best part is argument parsing and modules.
Shell is something that we all have started with, hence I describe it with words “omnipresence” or “has ubiquity”. Quick filtering or command chaining with Pipe (”|”) and CLI utilities like sed, awk, grep, etc.
Ajay: Awesome, let me give it a shot; it would certainly help adding new skills to my knowledge bank, at the very least.
Ajay- who has been a sysadmin all the while writing scripts in shell, just wants to try Python and see how he can make use of both to add more efficiency to his day-to-day work.
So, this is
Ajay’s review on linux academy’s course for Python which he found out while browsing the internet and it simply said: “Learn or start only with modules and structure that would be useful for you as Sysadmin and not the whole Python at once!”
Ajay can be any person who has spent a good time with shell, and not specially sysadmin. So this is for anyone who just wants to add an element to fun to work and has been wanting to learn Python for a long time!
Note: This is in no way a connection or recommendation from me to try only Linux Academy; I just loved this course and this is an honest review.
The following is an introduction to the course, where I’ll describe each section, add my inputs for code conversion, using specific modules and the outcomes!
Author: Keith Thompson
Getting started | Environment setup | Introducing Python
Keith has done a fantastic job of making sure your base is set up strong here. That means, setting up a cloud development server and taking care of small small things, like having a custom setup for
Customizing bash prompt: $ curl https://raw.githubusercontent.com/linuxacademy/content-python3-sysadmin/master/helpers/bashrc -o ~/.bashrc
Customizing Vim: $ curl https://raw.githubusercontent.com/linuxacademy/content-python3-sysadmin/master/helpers/vimrc -o ~/.vimrc
Now, these are completely optional you can have your own setup and can go with more advanced stuff like
the ultimate Vim configuration: https://github.com/amix/vimrc
Setting up and customizing stuff- I feel, sets the tone right away that Keith wants you to have fun customizing stuff and getting in the same tone as himself. This section ends with history & benefits of python along with the most important thing, that is the comparison between Python 2 and Python 3.
Running Python | Common Data Types | Control Flow
Now, you won’t jump into writing scripts right away, so the first step is to understand how python interprets stuff. REPL (Python Interactive Shell) is used here for rapid experimentation.
Common data types like Strings, numbers, Booleans, working with variables, lists, tuples, dictionaries.
Control Flow, mostly talks about loops, comparisons and logic operations. If you are like me who had started to learn python a lot many times and failed to move after the basic step this would be something you have already done. But, I would suggest to quickly go through this, as it could be a refresher.
Scripting with Python 3 is divided into 2 different parts and that’s the beauty of the course. Basic Scripting | Intermediate Scripting
It’s more about reading user inputs, functions, interacting with files and working with environment variables. Obviously all of these are important, although the super helpful part to me as a sysadmin was the topic on interaction with files.
Fun exercises in course: Write a script that does the following: 1. Takes a port_number as its only argument. 2. Calls out to lsof to determine if there is a process listening on that port. 3. If there is a process, kill the process and inform the user. 4. If there is no process, print that there was no process running on that port.
This is where you can start to see the heat of chapters to come, I mean they are not very complicated but it needs a lot of attention and practice.
Parsing command line parameters, Handling errors, Exit status, Execute shell commands with Python!
Yeah, that sounds familiar and interesting right? When I first saw the course structure, I was eagerly waiting for the part that said “Execute shell commands with Python”. It was fun learning the “subprocess” module.
>>> import subprocess >>> proc = subprocess.run(['ls', '-l’])
The proc variable is a CompletedProcess object, and this provides us with a lot of flexibility. We have access to the returncode attribute on our proc variable to ensure that it succeeded and returned a 0 to us.
>>> proc = subprocess.run( ... ['ls', '-l'], ... stdout=subprocess.PIPE, ... stderr=subprocess.PIPE, … )
>>> proc CompletedProcess(args=['ls', '-l'], returncode=0, stdout=b'total 0\ndrwxrwxr-x. 2 sachcode sachcode 6 Mar 12 13:53 bin\ndrwxrwxr-x. 3 sachcode sachcode 22 Apr 11 13:50 code\ndrwxrwxr-x. 5 sachcode sachcode 170 Apr 22 18:17 hr\ndrwxrwxr-x. 2 sachcode sachcode 50 Apr 11 12:39 install-scripts\ndrwxrwxr-x. 3 sachcode sachcode 223 Apr 10 19:32 sample\n', stderr=b'')
>>> proc.stdout b'total 0\ndrwxrwxr-x. 2 sachcode sachcode 6 Mar 12 13:53 bin\ndrwxrwxr-x. 3 sachcode sachcode 22 Apr 11 13:50 code\ndrwxrwxr-x. 5 sachcode sachcode 170 Apr 22 18:17 hr\ndrwxrwxr-x. 2 sachcode sachcode 50 Apr 11 12:39 install-scripts\ndrwxrwxr-x. 3 sachcode sachcode 223 Apr 10 19:32 sample\n'
>>> print(proc.stdout.decode()) total 0 drwxrwxr-x. 2 sachcode sachcode 6 Mar 12 13:53 bin drwxrwxr-x. 3 sachcode sachcode 22 Apr 11 13:50 code drwxrwxr-x. 5 sachcode sachcode 170 Apr 22 18:17 hr drwxrwxr-x. 2 sachcode sachcode 50 Apr 11 12:39 install-scripts drwxrwxr-x. 3 sachcode sachcode 223 Apr 10 19:32 sample
Now, that was a long talk about the subprocess module. I did that because I took a break from this course for 2 days and could see that subprocess was the bread and butter for us- the sys admins. There are some fun exercises in this section, so you’ll have fun please don’t skip them.
Standard: random & json | shutil & glob | re & math Third party: Using Pip & Virtualenv
pip is the reference Python package manager and
virtualenv is a tool that creates an isolated environment in Python.
$ pip install virtualenv $ pip list $ mkdir sample $ cd sample $ virtualenv sample_project $ source sample_project/bin/activate
The practical exercise(s) in this section is not something that you would want to skip, as they are fun!
🚀 It would be stuff like taking a simple URL in the script and parsing the output based on what anyone wants, say JSON or HTML.
Keith has been very frank in notifying that, the following section would be trickier and might need more attention; basically it won’t be easy stuff. And yeah it wasn’t, I had to do this part twice at least before I moved to the exercise to implement another project.
Project: We have many database servers that we manage, and we want to create a single tool that we can use to easily back up the databases to either AWS S3 or locally. We would like to be able to: > Specify the database URL to backup. > Specify a “driver” (local or s3) > Specify the backup “destination”. This will be a file path for local and a bucket name for s3. > Depending on the “driver”, create a local backup of the database or upload the backup to an S3 bucket.
Usage: .............. Pass in a full database URL, the storage driver, and destination. S3 Example w/ bucket name: $ pgbackup postgres://email@example.com:5432/db_one --driver s3 backups Local Example w/ local path: $ pgbackup postgres://firstname.lastname@example.org:5432/db_one --driver local \ /var/local/db_one/backups/dump.sql
It’s goes very deep in terms of setting up!
Test automation frameworks are famous for Python and majorly there are a couple of them that contribute to unit testing, like
Robot, Pytest, Unittest or PyUnit. In this course Keith uses Pytest as the framework.
It really depends on what you would like to use and here’s a good comparison of some of them.
Techniques like stubs and mocks can be used to separate code into “units” and run unit/individual level testing.
📓 Stubs and Mocks: They both are dummy objects that simulate fake methods that are being tested.
A Stub is used to fill in some dependency required for unit test to run correctly. A Mock on the other hand is a fake object which runs the tests where we put assert.
$ pgbackup --driver local ./local-dump.sql \ postgres://demo:email@example.com:80/sample $ pgbackup --driver s3 pyscripting-db-backups \ postgres://demo:firstname.lastname@example.org:80/sample
We’re going to write three tests to start: 1. A test that shows that the CLI fails if no driver is specified. 2. A test that shows that the CLI fails if there is no destination value given. 3. A test that shows, given a driver and a destination, that the CLI’s returned Namespace has the proper values set.
The course goes a but more deeper on the path of how we could implement the CLI guided by tests, how to write mocking tests, implementing PostgreSQL and local storage, and finally the AWS interaction.
As the logic says we sysadmins won’t be sharing or open sourcing every little tool we write, but we will want it to be distributable. The newest and preferred way to distribute a python tool is to build a ‘wheel’.
Wheel documentation: link
The final part of the course has a set of exercises that is again a full project implementation. Just adding a bit of introduction on same, and didn’t wanted to be a full spoiler here:
TDDyou will be writing tests, and building the
JSONparsing and exporting
console part, as we did in the above guided project.
This is the info with which Keith ends the course:
Now that you have a grasp of Python scripting, there are a lot of doors open to you to continue your learning. Dive into more configuration management using Ansible (which is written in Python), learn about data science, write more custom scripts, or maybe dive into web development.
That’s true, now this is not something in
Python, but I’ve found a good library of sysadmin scripts that you can either use the same way or try changing them and practicing in
Keith’s library for the course:
------- Helpful resources/references: -------