Using LogParser to Weed Through Call Manager Events

Author
William Bell
Vice President, Solutions and Products

The issues with Event logs on a Windows based CM host:

  • First, all of the events generated by CM services will use a generic event ID.  For example, 3 is the event code for all error level events.
  • Second, a single instance of an error such as ‘DeviceTransientConnection’ is quite common particularly in environments with a large number of VoWLAN phones or extended use of Extension Mobility.  Check out your logs for events around 9am and 9:30am to see what I mean.
  • Third, there is no event correlation mechanism in the cluster, so an event for a single phone leaving the network can be recorded in multiple event logs (depending on the nature of the event)

One tool that I use extensively to analyze log files (including Windows Event files) is the Microsoft LogParser (version 2.2) tool (http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en) .

This tool was put together by a former MS employee for performing diagnostics on Exchange servers but it can do much more than that.  The tool using a structured query language (SQL) syntax to filter, parse, and sort files in a variety of formats.  Formats include CSV or Tab delimited files, XML, IIS w3c log files, Windows event logs (in native format no less), and other variations.

I have used the tool to in various ways and it is not limited to the functions outlined in this blog.  The tool also has the ability to use a COM object so that your analysis can be scripted (on a Windows platform for instance) and it can export data into a CSV file, to the screen, in a data grid, or create charts and graphs.

Anyway, some examples are in order.

Scenario: You find that your Event log is flooded with error messages from the Call Manager service.  Let’s say it is a few hundred or even a few thousand messages.  You open a few of the events and you see that there are a mixture of transient connections and devices unregistering.

If you wanted to get a feel for the number of type of errors, you could run a query using logparser (why not, right):

logparser “select SUBSTR( Message, INDEX_OF(“Message”, ‘Error:’), INDEX_OF(“Message”, ‘-‘)) as ErrType, Count(Message) as CNT from *.evt group by ErrType” -i:evt

So, what is this doing?  First, notice that it is structure like a standard SQL select query (select … from … where …).  Also notice that you can embed functions directly in the select portion of the statement.  The SUBSTR() function is basically going to take a portion of the target string (Message in this case) starting at one point in the strong and ending at another.  Huh?

Using the following example:

Message is the body of the event message.  Since we don’t have the luxury of useful event IDs we have to try and figure out what we are dealing with by parsing the Message.

We use the SUBSTR() function to look for the term “Error:” and read it until we find the “-” (dash) character.  Most of the Cisco events will at least follow this format.

In the screen above, we are looking at a “DeviceTransientConnection” error.

The “as ErrType” portion of the select clause is just a name I picked as the column header for the output.   The Count() function will return the number of messages seen for each error type.  This relationship is made with the “group by ErrType” portion of the query.  The -i flag tells logparser what type of input to expect, in this case event logs.  Notice in the from clause we specify *.evt.  This is handy, logparser will take multiple event log files and compile them to a single output.

Since we didn’t specify an output type (using -o:) the output will go directly to the console:

That is quite a few TransientConnections to be sure.  Also a few other areas to be concerned about that you may have missed due to frustration with looking through the log file one event at a time.

Using LogParser, you can further drill into the DeviceTransientConnection errors by grouping transient errors by Device Name (helps find the top talkers) or by date/time which can help pinpoint a possible concentration of events based on time of day.

To sort by device name, you could use a query like:

logparser “select EXTRACT_TOKEN (Message, 2, ‘:’) as DevName, Count( Message ) as CNT from *.evt where message like ‘

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.