Connecting to Multiple Devices with Netmiko Using Python Threads and Queues

tl;dr – Click here to go straight to the Python example.

The journey to automation and scripting is fraught with mental obstacles, and one concept I continued to not really comprehend in Python was the concept of threading, multiprocessing, and queuing.

Python Logo

Up until recently, I felt like I basically had my dunce cap on (relatively speaking, of course) and was restricted to sequential loops and connections — in other words, I was stuck in “for i in x” loop land and could only connect to one device at a time. In order to speed up my scripts and connect to multiple devices at once (using Netmiko, for example), the path to that is through queues, and threading/multiprocessing.

Ultimately I landed on threading instead of multiprocessing because when you’re connecting to devices/APIs over the network, you’re typically waiting for a remote host to process the request, and thus your CPU is sitting there ‘idle’ waiting. To quote a great blog post that breaks down threading versus multiprocessing:

“[t]hreading is game-changing because many scripts related to network/data I/O spend the majority of their time waiting for data from a remote source. Because downloads might not be linked (i.e., scraping separate websites), the processor can download from different data sources in parallel and combine the result at the end.” (source)

While the above link has some great examples, for some reason I still didn’t quite grasp the concept of threads and queues, even after trying the example of other approaches. Why? Well, sometimes we need different perspectives to a problem because we all learn differently, thus my hope here is to provide a different perspective to threading and connecting to multiple devices with Python.

Netmiko Using Threaded Queue

I don’t want to waste to much time, so let’s just cut to the chase and get to the script:

I’m going to try a different approach here, so here’s an overly verbose perspective on how the script runs. It’s a step-by-step breakdown of how it processes. That said, as much as a I tried to describe the process in a linear manner, it’s not going to be perfect.

  1. Load and stage the modules and terminal messages relating to hitting ctrl+c. (lines 6-20)
  2. Load the global variables (lines 23-38)
    1. Prompt the user to securely enter a password (var: password) (line 23)
    2. Read a list of IP addresses from text file (vars: ip_addrs_file and ip_addrs) (lines 26-27)
    3. Set up the number of threads to spin up (var: num_threads) (line 30)
    4. Set up the global queue that we’ll use to set up a list of ip addresses to be processed later (var: enclosure_queue) (line 32)
    5. Set up an object that we’ll use to lock up the screen to print the contents of a thread so as to avoid having threads print over each other later (var: print_lock) (line 34)
    6. Set up the command you’ll want to run. This is a simple one command script for the purpose of the demo. (var: command) (line 38)
  3. The two functions deviceconnector() and and main() get loaded and staged. (lines 41-107)
  4. The main() function is called and begins the execution of the main components of the script (line 112)
    1. Loop through a number list (num_threads), and for each number in that list (i): (line 92)
      1. Load a thread that runs an instance of deviceconnector() sending to the function the thread number (i) and the global queue (enclosure_queue) (line 95)
        1. deviceconnector() accepts the variables i as i, and enclosure_queue as q (line 41)
        2. deviceconnector() starts an unending while loop that: (line 45)
          1. Attempts to acquire an IP address from the global queue (line 50)
            1. If there is no IP address in the queue, the while loop will be blocked and wait until there is an ip address in the queue
          2. Sets up dictionary for Netmiko (lines 54-59)
          3. Netmiko attempts to connect to the device (lines 52-53)
            1. If there is a time out, lock the process and print a time-out message, marking the queue item as processed and restarting the while loop (lines 55-58)
            2. If there is an authentication error, print an authentication error and exit the whole script (lines 70-73)
          4. Send command to the device, lock the process and print the output (lines 76-80)
          5. Disconnect from the device, mark the queue item as complete, and loop back (lines 83-86)
      2. Set the thread to run as a daemon/in the background (line 97)
      3. Start the thread (line 99)
    2. Loop through a list of IP addresses (ip_addrs), and for each IP address (ip_addr) (lines 102-103)
      1. Add the IP address (ip_addr) to the global queue  as an individual queue item to be processed (line 103)
    3. Wait for all queue items to be processed before exiting the queue and script (line 106)
    4. Print a statement to the console indicating the script is complete (line 107)

Use this as you wish, and hope it’s helpful.

Here’s the Github version.

Credit and Additional Info

This was inspired by a few different blog posts, so here’s some additional info to follow:

  • Multiprocessing Vs. Threading In Python: What You Need To Know.
    A great breakdown of threading versus multiprocessing, and influential for some of the work I’m doing.
  • How to Make Python Wait
    This one actually reignited my interest in figuring out how to use threading. It’s a good explanation of the different approaches to make a script wait in Python.
  • Queue – A thread-safe FIFO implementation
    Although written in Python 2, this post helped me put everything together so I could understand what the heck is going on. Some of the code I used here, but refactored for Python 3. Below is a crude diagram I did to help me figure out what was going on with this post, and the circles with arrows indicate loops, with the ‘f’ in the middle meaning ‘for’ loops and ‘w’ meaning ‘while’ loops.

Diagram that attempts to show how the thread and queue process works. Too complicated to explain in an alt tag, so look at code.

ztpgenerator: A ZTP Python Script for Juniper Devices (Maybe more?)

This is a script I’ve been working on for simplifying the ZTP process for Juniper switches.

https://github.com/consentfactory/ztpgenerator

What does it do?

  • Updates ISC-DHCP for ZTP devices (creates DHCP reservations)
  • Creates Juniper configurations based on Jinja2 templates
  • Can also create virtual chassis configurations if desired

There’s some pre-work that needs to be done for set up, but it’s fairly simple to deploy and doesn’t require a lot.

All of the documentation is there on Github.

Hoping this helps others out there.

Kirk Byers Python for Network Engineers Videos

Over the last three months I’ve been transitioning back over to network engineering. When I first started working in IT, my role was as a network technician, but slowly I took on more and more systems administration and engineering, and the work I was doing was mostly with PowerShell.

As a result, my Python skills have atrophied over the years since I was so focused on PowerShell, and if there’s one thing I’ve learned while being put in charge of managing thousands of switches, it’s that I need some scripting in place to handle mass changes.

 

Python Logo

The perfect class to get myself back in the Python game was from reddit, offered by Kirk Byers called “Python for Network Engineers“. It’s a introductory course on some Python basics, which for offered enough to get me going again.

I’m putting this list of the videos online for myself since I hate referencing my emails all the time, but if someone else finds it useful, awesome.

Lesson 1: Why Python, the Shell, and Strings
Lesson 2: Numbers, Files, Lists, and Linters
Lesson 3: Conditionals and Loops
Lesson 4: Dictionaries, Exceptions, and Regular Expressions
Lesson 5: Functions and the Python Debugger
Lesson 6: Netmiko
Lesson 7: Jinja2, YAML and JSON
Lesson 8: Libraries, PIP, and Virtual Environments

Lesson 1: Why Python, the Shell, and Strings

  1. Introduction 
    Video https://vimeo.com/243034300
    Length is 7 minutes
  2. Why Learn Programing?
    Video https://vimeo.com/243905715
    Length is 1 minute
  3. Why Python?
    Video https://vimeo.com/243909371
    Length is 3 minutes
  4. Python2 versus Python3
    Video https://vimeo.com/243912631
    Length is 2 minutes
  5. Characteristics of Python
    Video https://vimeo.com/243918300
    Length is 5 minutes
  6. The Python Interpreter Shell
    Video https://vimeo.com/242411259
    Length is 9 minutes
  7. IPython
    Video https://vimeo.com/242460561
    Length is 4 minutes
  8. Printing to stdout and Reading from stdin
    Video https://vimeo.com/243028886
    Length is 6 minutes
  9. Dir, Help, and Variables
    Video https://vimeo.com/243480156
    Length is 10 minutes
  10. Python Strings (Part 1)
    Video https://vimeo.com/243481392
    Length is 6 minutes
  11. Python Strings (Part 2)
    Video https://vimeo.com/243482081
    Length is 8 minutes
  12. Python Strings (Part 3)
    Video https://vimeo.com/243482871
    Length is 10 minutes
  13. Python String Formatting (Part 1)
    Video https://vimeo.com/243936489
    Length is 12 minutes
  14. Python String Formatting (Part 2)
    Video https://vimeo.com/243956669
    Length is 4 minutes

Lesson 2: Numbers, Files, Lists, and Linters

  1. Numbers
    Video https://vimeo.com/244128549
    Length is 9 minutes
  2. Files
    Video https://vimeo.com/244127459
    Length is 10 minutes
  3. Lists
    Video https://vimeo.com/244257596
    Length is 6 minutes
  4. List Slices
    Video https://vimeo.com/244259492
    Length is 4 minutes
  5. Lists are Mutable
    Video https://vimeo.com/244287000
    Length is 5 minutes
  6. Tuples
    Video https://vimeo.com/244153105
    Length is 3 minutes
  7. Using .join()
    ​Video https://vimeo.com/245464488
    Length is 3 minutes
  8. sys.argv
    Video https://vimeo.com/245464766
    Length is 2 minutes
  9. Linters
    Video https://vimeo.com/245102246
    Length is 6 minutes

Lesson 3: Conditionals and Loops

  1. Conditionals
    Video https://vimeo.com/245104620
    Length is 8 minutes
  2. Boolean Logic (Booleans, Ternary Operator, None)
    Video https://vimeo.com/245112558
    Length is 8 minutes
  3. Python For Loops
    Video https://vimeo.com/245466297
    Length is 5 minutes
  4. For Loops (Enumerate)
    Video https://vimeo.com/245477015
    Length is 6 minutes
  5. For Loops (Break and Continue)
    Video https://vimeo.com/245478016
    Length is 9 minutes
  6. While Loops
    Video https://vimeo.com/245545155
    Length is 5 minutes
  7. Loops Miscellaneous
    Video https://vimeo.com/245552604
    Length is 6 minutes

Lesson 4: Dictionaries, Exceptions, and Regular Expressions

  1. Dictionaries
    Video https://vimeo.com/246157566
    Length is 6 minutes
  2. Dictionaries Methods
    Video https://vimeo.com/246163031
    Length is 7 minutes
  3. Sets
    Video https://vimeo.com/246167477
    Length is 9 minutes
  4. Exceptions
    Video https://vimeo.com/246174686
    Length is 15 minutes
  5. Regular Expressions (Part1)
    Video https://vimeo.com/246184715
    Length is 15 minutes
  6. Regular Expressions (Part2)
    Video https://vimeo.com/246532117
    Length is 7 minutes
  7. Regular Expressions (Part3)
    Video https://vimeo.com/246534450
    Length is 8 minutes
  8. Regular Expressions, Other Methods
    Video https://vimeo.com/246535038
    Length is 4 minutes

Lesson 5: Functions and the Python Debugger

  1. Functions (Part1)
    Video link https://vimeo.com/247570174
    Length is 8 minute
  2. Functions (Part2)
    Video link https://vimeo.com/247581011
    Length is 11 minutes
  3. Misc Topics (Part1)
    Video link https://vimeo.com/247582360
    Length is 10 minutes
  4. Misc Topics (Part2)
    Video link https://vimeo.com/247655574
    Length is 8 minutes
  5. Python Debugger (pdb)
    Video link https://vimeo.com/247724017
    Length is 10 minutes

Lesson 6: Netmiko

  1. Netmiko Introduction and Basics 
    Video link https://vimeo.com/254569911
    Length is 8 minutes
  2. Netmiko Show Commands
    Video link https://vimeo.com/254578980
    Length is 13 minutes
  3. Netmiko and Prompting
    Video link https://vimeo.com/254587832
    Length is 12 minutes
  4. Netmiko and TextFSM
    Video link https://vimeo.com/254611876
    Length is 10 minutes
  5. Netmiko Config Changes
    Video link https://vimeo.com/254614073
    Length is 8 minutes
  6. Netmiko Troubleshooting
    Video link https://vimeo.com/254786724
    Length is 9 minutes

Lesson 7: Jinja2, YAML and JSON

  1. Jinja2 Basics
    Video link https://vimeo.com/257997257
    Length is 7 minutes
  2. Jinja2 For-Loops and Conditionals
    Video link https://vimeo.com/257999160
    Length is 9 minute
  3. Jinja2 and CSV
    Video link https://vimeo.com/258142987
    Length is 5 minutes
  4. Jinja2 Dictionaries and Nested Loops
    Video link https://vimeo.com/258145504
    Length is 11 minutes
  5. YAML Basics
    Video link https://vimeo.com/258161182
    Length is 9 minutes
  6. YAML Part2
    Video link https://vimeo.com/258169427
    Length is 10 minutes
  7. Using Python to Write YAML
    Video link https://vimeo.com/258171559
    Length is 3 minutes
  8. JSON
    Video link https://vimeo.com/258178243
    Length is 5 minutes
  9. Managing Data Structures
    Video link https://vimeo.com/258181273
    Length is 5 minutes

Lesson 8: Libraries, PIP, and Virtual Environments

  1. Importing Libraries
    Video link https://vimeo.com/259422351
    Length is 5 minutes
  2. sys.path and PYTHONPATH
    Video link https://vimeo.com/259423316
    Length is 7 minutes
  3. pip
    Video link https://vimeo.com/259424573
    Length is 7 minutes
  4. Virtual Environments
    Video link https://vimeo.com/259426537
    Length is 6 minutes
  5. Creating a Simple Python Module
    Video link https://vimeo.com/259427586
    Length is 4 minutes