Quickie: Viewing Transceivers Inserted into Palo Alto Firewalls and Other Transceiver Bits

If you’re running a Palo 5200/7000 series firewall, you may have inserted transceiver modules and the only way you really know they’re recognized is if you get them online. While troubleshooting an issue, I found a item from the Palo Alto KB, namely the log file brdagent.log is where you can view the transceiver you inserted; the log itself is for monitoring interfaces and other networking chips being inserted (see book link below).

To view the log, I prefer to use the tail command to view the log, which is a Unix/Linux command to view the last section of the log file, and even to watch the log file as it’s being used.

The command:

tail lines 100 follow yes cp-log brdagent.log
<bunch of log files>
cp brdagent.log 2019-10-03 17:10:40 2019-10-03 17:10:40.001 -0600 Port 25: Down 40Gb/s-full duplex
cp brdagent.log 2019-10-03 17:10:40 2019-10-03 17:10:40.001 -0600 PORT25: board_port_autoneg_enabled -> board_port_autoneg, link: 0, mode: 1
cp brdagent.log 2019-10-03 17:11:40 2019-10-03 17:11:40.644 -0600 Error: gryphon_get_port_info(5200/gryphon_ports.c:1231): Remote fault detected on port 25!
cp brdagent.log 2019-10-03 17:13:22 2019-10-03 17:13:22.156 -0600 Port 25: sysd_obj_lookup for <QSFP> failed !!!
cp brdagent.log 2019-10-03 17:13:22 2019-10-03 17:13:22.156 -0600 Port 25: Empty QSFP+ detected
cp brdagent.log 2019-10-03 17:14:07 2019-10-03 17:14:07.441 -0600 QSFP+ detected on port 25
cp brdagent.log 2019-10-03 17:14:07 vendor 'FINISAR CORP. '; part 'FTL4E1QE1C-A5 '; id 'QSFPP'

The command broken down:

  • tail – the Unix/Linux command that displays the last section of a file
  • lines 100–  displays the last 100 lines (or less) of the file
  • follow yes – informs the cli to continue printing any additional input to the file
  • cp-log– this is a directory for control plane logs, and according to Tom Piens with the Mastering Palo Alto Networks book below, there a few different directories for the management plane, data plane, and other logs. The log directories will vary based on models.
  • brdagent.log– the log filename.

As you see in the log above, you can see the the QSFP transceiver in the log. Using the tail follow command though, you can watch as transceivers are inserted, like this (I inserted a Cisco twinax as an example):

2021-05-25 13:50:34.663 -0600 Port 19: SFP Plus Empty detected
2021-05-25 13:50:51.756 -0600 Port 19 :: sfp_data.transceiver_codes(str) = 10000
2021-05-25 13:50:51.756 -0600 SFP Plus detected on port 19
vendor 'CISCO-TYCO '; part '2053783-3 '; tc '10000'
2021-05-25 13:50:51.757 -0600 Port 19: SFP Plus Fiber detected
2021-05-25 13:50:51.757 -0600 PORT19: board_port_sfp_nopop_0 -> board_port_startup, link: 0, mode: 2
2021-05-25 13:50:51.761 -0600 PORT19: board_port_startup -> board_port_autoneg, link: 0, mode: 2
2021-05-25 13:50:51.764 -0600 PORT19: board_port_autoneg -> board_port_autoneg_linked, link: 1, mode: 2
2021-05-25 13:50:51.764 -0600 Port 19: Up 10Gb/s-full duplex

I actually found a pretty good book that I purchased which breaks this down some of these commands. Mastering Palo Alto Networks: Deploy and manage industry-leading PAN-OS 10.x solutions to secure your users and infrastructure explains how to view some of these logs, particularly the section on ‘Debugging Processes’.

Other Transceiver Bits

Some additional bits I’ve learned recently:

  • Yes, you can other vendor/third-party transceivers in Palo Alto firewalls. Palo Alto publishes the details of the transceivers here. I haven’t ever encountered issues with other vendor transceivers, but that said, getting support if you have an issue and the transceiver is potentially the problem may not result in support from customer support.
  • While troubleshooting an HSCI port, I found that I could test if the optic was functioning correctly from the HSCI connecting the port to a switch and configuring layer 2 connectivity (ethernet). The transceiver responded, which makes sense because the configuration in HA sets the transport for HSCI as ethernet. Even though Palo says, “[t]he traffic carried on the HSCI ports is raw Layer 1 traffic, which is not routable or switchable. Therefore, you must connect the HSCI ports directly to each other (from the HSCI port on the first firewall to the HSCI port on the second firewall),” I found the HSCI at least responds to ethernet. So perhaps they mention this to indicate you shouldn’t bridge the connection (via layer 2 switch).
  • If you’re not getting complete session sync between a pair of HA firewalls, and even though your HA is showing as up, you should perhaps check your transceivers, particularly if you’re using non-Palo Alto transceivers.

Personal KB Reference:

Quickie: Setting Up SMTP Relay With G Suite Domain Without Pulling Your Hair Out and Questioning Your Google-Fu Skills

For the love of Pete!

I just spent an hour or two in a furious state trying to set up SMTP relay with a G Suite domain, and it was stupidly frustrating because there was one little component stopping everything. Oh the rage!

Kid in Incredible Hulk outfit raging...kind of.

My intent was simple: SMTP relay using a domain account for authentication and TLS. Should be easy.

When you look at the the instructions for setting up SMTP relay, they do appear on the surface pretty straight-forward:

  • Go to G Suite > Apps > Gmail > Advanced Settings
  • Make sure you’re at the top level of the OU structure (you should be)
  • Add “SMTP relay service” (it gives you a few options, in my case I want to use an account to authenticate, see below)

SMTP Relay service options in G Suite

Then you configure your SMTP relay settings on your application to point to smtp-relay.gmail.com on port 587, input your SMTP authentication and then all done, right?

WRONG! You’re going to get continual authentication errors (Google’s SMTP error code “535 5.7.8”) and Google’s SMTP service will tell you to pound sand and send you to unhelpful help articles about having bad credentials.

You have to perform one more step not mentioned in the Google documentation (here’s where I found this fix): enable “Less secure app access” in your Google service SMTP account settings. Easiest way to get to it is go to Gmail > click on the service account profile > click ‘Manage your Google Account’ (tangent: why is ‘your’ not capitalized?) > then just search on the top for ‘Less secure app access’ and toggle the button to on. It looks like this:

Google account settings with Less Secure App toggle buttonThat’s it! After that, SMTP relay will start working correctly.

Maybe this will help prevent some Hulk transformations.

(Update 20191220.1238 – Reworked some parts because I was in a rush last night).