Audit Logs on Cisco Unified Communications Manager (CUCM)

Author
William Bell
Vice President, Solutions and Products

Background

I will keep the background discussion to a minimum, because I am sure most readers of this blog understand the need for an auditing solution.  To summarize, admins need a way to look at an audit trail for administrative changes that occur in their CUCM cluster.  Basically, in large organizations there is a need to look at “what has changed” so that if there is an issue reported that is related to human error, the error can be quickly identified and fixed.  There is also an accountability aspect as it is likely that senior support folks want to identify education gaps in their lower tier support teams.

Software Revisions

The ability to enable auditing for administrative tasks was included in CUCM version 7.1(2) and later.  In my lab, I tested this feature using a CUCM 7.1(3b)SU2 cluster.

Enabling Audit Logs

By default, audit logs for administrative level tasks are enabled.  To access this feature, you will need to browse the audit log parameters from Cisco Unified Serviceability.  The following procedures can be used to accomplish this task:

1. Connect to Cisco Unified Serviceability on your CUCM cluster: https://mypublisher/ccmservice/

2. Go to Tools>Audit Log Configuration

3. Depending on your particular operational model, you may have people with different levels of authorization.  Authorization levels are implemented via the Groups and Roles configuration objects.  If you have a need to customize roles or Group/Role assignments you will want to look at the “Standard Audit Log Administration” role to understand the level of access required for users/groups that need to look at audit logs.

Accessing Audit Logs

At this point, you have enabled audit logs for administrative tasks.  As with other logs and trace files on the system, you can access these logs in several ways.  For this example, we will walk through using RTMT to view audit logs.

1. Load RTMT on your admin workstation

2. Connect to your CUCM cluster using a user ID that has the “Standard Audit Log Administration” role (or equivalent) assigned.

3. Go to System>Tools>Trace>Trace and Logs Central

4. Under “Trace and Logs Central”, you will see an “Audit Logs” folder.  Double click the folder to start the log collection wizard.

One of the benefits in using RTMT is that you can collect and download AuditLogs either as a one time request or you can schedule the download.  For those that like to run pro-active analysis (via scripts and the like) you may want to look at the “Schedule Download” option.  As with other traces/logs, once you specify the search criteria RTMT will provide a folder tree structure that can be used to navigate to the audit logs stored on the server.

 

Deciphering the Audit Log

The log file entries are lengthy but parsing them is not at all difficult.  Let’s look at an example.  I have a dummy phone on my CUCM cluster.  The phone was already created prior to testing the audit log feature.  What I did was add a phone number to the device.  In the AuditLog this action is captured as follows:

04/06/2010 11:03:15.295 |LogMessage
UserID :ccmadministrator
ClientAddress :192.168.1.4
Severity :5
EventType :GeneralConfigurationUpdate
ResourceAccessed:CUCMAdmin

EventStatus :Success
AuditDetails : record in table numplan with key field dnorpattern = 2025552222 added
ComponentID :Cisco CUCM Administration
App ID:Cisco Tomcat
Cluster ID: Node ID:iecucm01

Now, the log entry itself will be a single line in the trace file but I have broken it out in something a tad more legible.  You will see that the log identifies the user who performed the change (UserID), the IP address of the machine the user was using (ClientAddress) along with other pertinent information.  Of particular interest is the AuditDetails section of the logged event.  In this example, we can see that I added a directory number.   Before dissecting this section, we should address an obvious question: which device was this performed on?

This is where things are tricky.  If you look at previous log entries, you will find when the user (ccmadministrator in our example) connected to the CCMadmin web portal.  However, the AuditLog will not track this user’s movements through the various web pages.  So, if I clicked on a device, and then clicked on “Add new line”.  Those events are not recorded.  Only when I actually “save” the change will it be recorded in the AuditLog.

Why?  Well, it boils down to the fact that the Audit Log is tracking changes to the database.  It’s focus is on what has changed in the database, not what has changed in the admin interface.  This is actually a necessary distinction, since changes to the database can come from a variety of sources:  BAT, CCMAdmin, AXL/SOAP (e.g. UCCX creates a JTAPI trigger).  If you look at our example above, you will see that the AuditDetails says:

record in table numplan with key field dnorpattern=2025552222 added

Numplan is a database table and if we wanted to find out what device this number was added to, we would need to take a few extra steps.  One step could be to run an ad hoc query from the command line to see what gives:

admin:run sql select d.name as device,n.dnorpattern,dmap.numplanindex
from numplan as n inner join devicenumplanmap as dmap on dmap.fknumplan=n.pkid
inner join device as d on dmap.fkdevice=d.pkid
where n.dnorpattern='2025552222'
device dnorpattern numplanindex
=============== =========== ============
SEP00258418EE10 2025552222 3

From this output, we have identified the device and the line appearance the new number is assigned.  Yeah, I know the query is ugly.  I can’t resist you see, I do most of my admin tasks via custom queries.  That’s just me.  You could certainly figure this out via the CCMAdmin web pages by going to Call Routing>Route Plan Report.  Then search for the directory number “2025552222”.

So, you still have some work to do when you want to analyze the audit log, but the data now available is a great improvement over what you would have to do in the past.  I won’t even waste your cycles on that discussion.

Other Things That Are Logged

As you play around with the Audit Log, you will find that it captures other interesting things as well.  Such as commands entered by users on the CLI.  This is handy if you want to find out who restarted the TomCat service or who reloaded the server.  Small things like that.  Of course, if you want that data to have any meaning, you may want to investigate the CLI command: set account name.

The Audit Log will also capture events like a user uploading a custom banner page to the CUCM system.  The custom banner page is another new security feature added to CUCM 7.1(2).

Where is the Audit Log

For those folks who don’t dig on using RTMT for certain tasks, and would just rather download the file or view it directly on the console I have two pieces of information:

1. The files are stored here:  activelog audit/AuditApp/*

2. If you go this route, I recommend downloading the files to your workstations and then viewing/parsing/filtering.  Especially during production hours.

Conclusion

Well, the AuditLog feature isn’t 100% perfect as you still have to tear yourself away from CNN or ESPN to actually do some work.  But, it is getting there and the information that is recorded in the AuditLog is very handy when you need to determine the “how”, “when”, and “why”.  Hmmm, maybe it won’t tell you the “why” but it sure will point you to the person that can!

10 responses to “Audit Logs on Cisco Unified Communications Manager (CUCM)

  1. Appreciate your detailed and insightful tutorial. It’s very timely because I’ve been searching on and off for the last few weeks for a method of auditing changes to a 6.0(2) cluster. I was wondering if there may be a way of tracking or capturing tomcat login and page serves for a specific user account. Can you think of a trace or capture that might accomplish this?
    Humbly,
    Steve Sutton

  2. Steve,

    Sorry for the delay in getting back to you. I missed this comment in my e-mail stream. The audit logs outlines in the article can be used for this purpose. Back in the day, I would use the IIS logs (pre-appliance model) to do something similar but that was a chore. You can see similar data with tomcat. On the CUCM system (usually the publisher node) you can view a daily access record for the tomcat service:

    [b]admin:file list activelog tomcat/logs/localhost* detail[/b]

    You can then view one of these files to see if it is close to what you need. I am not sure if there is any more value in tomcat that the audio logs don’t already provide but you can determine what best meets your requirements. (NOTE: You can also browse the tomcat logs using the same methods described in this article).

    Now, if you are looking for a way to pull or push the traces off box and then post process using perl or something similar, then you have a few hoops to jump through. You can push tomcat traces to a syslog server. Take a look at the CUCM Serviceability configurations (Alarm configurations). Tomcat traces can be configured under the platform services for your server.

    I am not sure if there is a way to push auditlogs to a syslog server but you can pull them on a scheduled basis using RTMT. There are other methods, but using RTMT is pretty straight forward.

    Regards,
    Bill

  3. Hi William,

    Actually this article is incorrect. The logs you are enabling above are the database logs and are filed under the informixauditlogs on RTMT. Application audit logs are enabled by default and are the checkboxes listed in the Application Audit Log Settings above.

    Cheers
    Guy Kelly

  4. Guy,

    Yup, you are absolutely correct. I have nothing to say except thanks. I have made the correction and plan to come back and expand upon blog later.

    Thanks,
    Bill

  5. Hi!
    Thank you for this very usefull article!
    Is there any way to delegate to any user a permission to manage only some pool of phones?
    In large organisation with many Regions anf offices we have to give to local admins roles:
    Standard CCM end users
    Standard CCM phone management
    Standard CCM user management
    Every thing works fine, local admin can add and change phones in his office.
    The problem that ye can can change and delete ANY phone in Organization.

    Is there any way to resolve it?
    Thanks
    Vladimir

  6. Vladimir,

    The behavior you are observing is by design. The CUCM authorization model only provides for what web pages you can access and the permission level on those pages. There is no way you can provide for a mult-tenant admin model using the native CUCM software.

    There are third-party applications that address this need. I haven’t looked at Cisco’s Unified Provisioning Manager, but it may provide what you are looking for (and then some). You could also look at Unimax, 2Ring, and similar partners. I cannot recommend a solution on this site. Perhaps query: "cucm multi-tenant provisioning" or "cucm multi-tenant administration" in Google.

    HTH.

    Regards,
    Bill

  7. commenting on response of :William Bell, February 24, 2011 ,

    Really this is good info ,,,pls include some examples sql commands ,
    i appreiciate for good work 🙂
    🙂

  8. Nice Post.. Excellent Info.. Really amazing.. This was a fantastic article… really superb….

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.