Python TTP Template for Cisco ASA Configuration Files

When working with Cisco ASA configurations and with scripting/automation, you know that ASA configuration files are not written with structured data in mind. A great solution to this is to use a Python tool called Template Text Parser, or TTP for short, which is networking/systems focused Python library that allows you to create structured data (JSON or YAML) from semi-structured configuration files or from the output of CLI show commands.

More information about TTP can be found on the link above. If you want a great tutorial on TTP, Kirk Byers has a great series on TTP.

Below is a TTP template that I created to do the following:

  • Create structured data for objects and object groups
  • Includes network, service, and protocol groups
  • Creates structured data for ASA ACLs

The end goal for this template is to migrate configurations from ASA to Palo Alto PANOS, creating set commands for equivalent configurations. However, this template is only focused on creating structured data from ASA firewall rules (ACLs).

One note: ACL lines in ASA configuration files can be made up of likely a few hundred variations or more, particularly if you have logging/debugging on. This template includes most of the more common ACL configuration variations.

pyribbon: Python Module for Sonus/Ribbon SBC REST API

Part of my recent stumbling into supporting VoIP systems again is supporting and maintaining Ribbon session border controllers (SBCs). SBCs are pretty much gateways/routers and even firewalls for VoIP traffic, which not only routes/limits traffic between disparate voice systems (like ISDN lines to SIP trunks), but can also manipulate the traffic such as adjusting the phone number format, media codec, changing SIP header information, yada yada — you get the point.

Diagram showing Ribbon SBC connecting to different systems

Pretty much all SBCs are configured via extensive web interfaces with lots of deep options, so if you have to manage these at scale, it can get a little time consuming — which is what my problem was. I have many SBCs scattered across our state, and I generally loathe having to make mass changes via any GUI, so does Ribbon have some method of configuring SBCs at scale? Yes, through its Ribbon API.

However, I’m not going to sit down and use Postman to configure these (well, maybe initially to test), so I need a scripting framework. I could use PowerShell or curl with bash scripting to do this, but most of my work in networking is done via Python, and thus because I couldn’t find any Python module built for this already (and it seemed like fun to create), I made this simple Python module and it’s on Github.

pyribbon: Python Module for Sonus/Ribbon SBC REST API

I wrote this with a VoIP engineer in-mind, and the basic idea is you import the module in a Python script, then interact with source of data to make changes or perform actions. For example, I’ve used it to create a ton of phone number changes with the routing and transformation tables, and I’ve used it to perform configuration backups across all systems.

To do most of this though, you need to get familiar with API and how object resources are designed. Personally I started with Postman and got a sense for how the API works, then I built the class to handle the API. Initially I started small, but then realized it was rather simple to add additional methods to the pyribbon class to perform different tasks. One could easily build additional methods such as reboot, backup, etc., but I just didn’t see the need for that — yet.

One potential cool use is to perhaps use the module to query for call statistics and send that to a database, then maybe graph it out with Grafana; or, if your NMS allows for script pollers, perhaps use that versus digging through SNMP OIDs.

Final thoughts: I know this is probably high on the esoteric use-cases. I mean, how many VoIP engineers are there in the world, then narrow that down to Sonus/Ribbon VoIP engineers, then dwindle that down to those that interact with the API, and then dwindle that down even more that do it via Python. Pretty small, but if you like Python for this kind of work, this should help.

Also, I thought about putting this in PyPi, but I can’t imagine this getting much use or further development on my end (it does what I need it to do well), and thus I’m shying away from that.

Let me know if you find this useful or have questions.

Connecting to Multiple Devices with Netmiko Using Python Threads and Queues

tl;dr – Click here to go straight to the Python example.

The journey to automation and scripting is fraught with mental obstacles, and one concept I continued to not really comprehend in Python was the concept of threading, multiprocessing, and queuing.

Python Logo

Up until recently, I felt like I basically had my dunce cap on (relatively speaking, of course) and was restricted to sequential loops and connections — in other words, I was stuck in “for i in x” loop land and could only connect to one device at a time. In order to speed up my scripts and connect to multiple devices at once (using Netmiko, for example), the path to that is through queues, and threading/multiprocessing.

Ultimately I landed on threading instead of multiprocessing because when you’re connecting to devices/APIs over the network, you’re typically waiting for a remote host to process the request, and thus your CPU is sitting there ‘idle’ waiting. To quote a great blog post that breaks down threading versus multiprocessing:

“[t]hreading is game-changing because many scripts related to network/data I/O spend the majority of their time waiting for data from a remote source. Because downloads might not be linked (i.e., scraping separate websites), the processor can download from different data sources in parallel and combine the result at the end.” (source)

While the above link has some great examples, for some reason I still didn’t quite grasp the concept of threads and queues, even after trying the example of other approaches. Why? Well, sometimes we need different perspectives to a problem because we all learn differently, thus my hope here is to provide a different perspective to threading and connecting to multiple devices with Python.

Netmiko Using Threaded Queue

I don’t want to waste to much time, so let’s just cut to the chase and get to the script:I’m going to try a different approach here, so here’s an overly verbose perspective on how the script runs. It’s a step-by-step breakdown of how it processes. That said, as much as a I tried to describe the process in a linear manner, it’s not going to be perfect.

  1. Load and stage the modules and terminal messages relating to hitting ctrl+c. (lines 6-20)
  2. Load the global variables (lines 23-38)
    1. Prompt the user to securely enter a password (var: password) (line 23)
    2. Read a list of IP addresses from text file (vars: ip_addrs_file and ip_addrs) (lines 26-27)
    3. Set up the number of threads to spin up (var: num_threads) (line 30)
    4. Set up the global queue that we’ll use to set up a list of ip addresses to be processed later (var: enclosure_queue) (line 32)
    5. Set up an object that we’ll use to lock up the screen to print the contents of a thread so as to avoid having threads print over each other later (var: print_lock) (line 34)
    6. Set up the command you’ll want to run. This is a simple one command script for the purpose of the demo. (var: command) (line 38)
  3. The two functions deviceconnector() and and main() get loaded and staged. (lines 41-107)
  4. The main() function is called and begins the execution of the main components of the script (line 112)
    1. Loop through a number list (num_threads), and for each number in that list (i): (line 92)
      1. Load a thread that runs an instance of deviceconnector() sending to the function the thread number (i) and the global queue (enclosure_queue) (line 95)
        1. deviceconnector() accepts the variables i as i, and enclosure_queue as q (line 41)
        2. deviceconnector() starts an unending while loop that: (line 45)
          1. Attempts to acquire an IP address from the global queue (line 50)
            1. If there is no IP address in the queue, the while loop will be blocked and wait until there is an ip address in the queue
          2. Sets up dictionary for Netmiko (lines 54-59)
          3. Netmiko attempts to connect to the device (lines 52-53)
            1. If there is a time out, lock the process and print a time-out message, marking the queue item as processed and restarting the while loop (lines 55-58)
            2. If there is an authentication error, print an authentication error and exit the whole script (lines 70-73)
          4. Send command to the device, lock the process and print the output (lines 76-80)
          5. Disconnect from the device, mark the queue item as complete, and loop back (lines 83-86)
      2. Set the thread to run as a daemon/in the background (line 97)
      3. Start the thread (line 99)
    2. Loop through a list of IP addresses (ip_addrs), and for each IP address (ip_addr) (lines 102-103)
      1. Add the IP address (ip_addr) to the global queue  as an individual queue item to be processed (line 103)
    3. Wait for all queue items to be processed before exiting the queue and script (line 106)
    4. Print a statement to the console indicating the script is complete (line 107)

Use this as you wish, and hope it’s helpful.

Here’s the Github version.

Credit and Additional Info

This was inspired by a few different blog posts, so here’s some additional info to follow:

  • Multiprocessing Vs. Threading In Python: What You Need To Know.
    A great breakdown of threading versus multiprocessing, and influential for some of the work I’m doing.
  • How to Make Python Wait
    This one actually reignited my interest in figuring out how to use threading. It’s a good explanation of the different approaches to make a script wait in Python.
  • Queue – A thread-safe FIFO implementation
    Although written in Python 2, this post helped me put everything together so I could understand what the heck is going on. Some of the code I used here, but refactored for Python 3. Below is a crude diagram I did to help me figure out what was going on with this post, and the circles with arrows indicate loops, with the ‘f’ in the middle meaning ‘for’ loops and ‘w’ meaning ‘while’ loops.

Diagram that attempts to show how the thread and queue process works. Too complicated to explain in an alt tag, so look at code.

ztpgenerator: A ZTP Python Script for Juniper Devices (Maybe more?)

This is a script I’ve been working on for simplifying the ZTP process for Juniper switches.

What does it do?

  • Updates ISC-DHCP for ZTP devices (creates DHCP reservations)
  • Creates Juniper configurations based on Jinja2 templates
  • Can also create virtual chassis configurations if desired

There’s some pre-work that needs to be done for set up, but it’s fairly simple to deploy and doesn’t require a lot.

All of the documentation is there on Github.

Hoping this helps others out there.