Network Stability Through Resilience Engineering
Everyone who has attempted to load a CUCM or CUC web page has seen that scary looking warning saying, “There is a problem with this website’s security certificate.”
Ahead Thar Be Dragons
On the page displaying the warning, the user is presented with the choice to “Click here to close this web page,” which it seems is the safest route, or to “Continue to this website (not recommended),” which is obviously an option that is foolish and reckless. Think James Dean drag-racing at 80 mph toward the cliff and Sal Mineo holding his breath to see who jumps out of the car first.
Most people who have worked with CUCM for any length of time don’t let the warning deter them. They charge ahead with a live-fast-die-young attitude, and if their sleeve gets caught in the door handle as they’re trying to open the sedan and they plunge to their death, oh well.
But after a while it can be kind of a pain to constantly have to click through the extra screen, so many administrators add the web certificate to the trusted store on their computer. This method works for many folks.
However, I recently became engaged by a client that wanted to have users employ the ccmuser and ciscopca web applications to manage their phones and voicemail accounts. The client is a government organization that is (naturally) very concerned about security, and their users have been instructed to never go to sites that flash warnings like the one above. They certainly are not allowed to add certificates to their trusted stores.
The way around this is, of course, to add a trusted certificate to CUCM and CUC that the Tomcat service can use when starting up the website. By “trusted” I mean that the certificate will already be in the store of the computers in question. This usually means that the certificate will be one that’s generated by a certificate authority (CA) server on the premises, or it means that a certificate will be purchased from some third party CA (like VeriSign or Go Daddy).
What we did was get the CA at the organization to generate a certificate for us.
The process of getting a new Tomcat certificate for the devices begins with generating a Certificate Signing Request (CSR) that can be given to the CA to ask for a certificate. This is done the same way in both CUCM and CUC.
One thing worth mentioning at this point is the fact that we’ve heard that there are more and more internal CAs that are using fraud protection. The protection frequently requires that the Cisco server administrator give the FQDN of their server to the person running the CA, and receive from the CA a number. The CSR has to be constructed to use the number instead of the FQDN of a server. When the CA processes the CSR, it takes the number mentioned therein, matches it to the name in the CA database, and generates a certificate with the correct FQDN. Beginning with CUCM v.9.x it’s possible to manipulate the information contained in the CSR, but not with versions before that. If you’re using CUCM v.8.x or below and your CA requires you to change your CSR, you’re going to have to find a new CA.
Assuming, however, that you’re using v.9.x or above (and I am), you’ll be able to do the required manipulation. You’ll also be able to do multi-server certificates which allow you to get a single certificate for your entire cluster (including the IM&P servers), upload it to your Publisher, and have it replicate to all of your servers. This is the method I’m going to describe.
To generate a CSR, follow these steps:
When the certificate is received back from the CA, it may have one of several possible extensions. It might even just be a text file that you have to copy and paste into notepad and save yourself. Interestingly, you can change the extension of the certificate to allow yourself to do different things with it. For instance, if you change the extension to “.pem” you can open the certificate using something like notepad and verify that the certificate really matches the CSR. To do this go back to SSLShopper, but this time use this link. Under “What to check,” select “Check if a CSR and Certificate Match,” copy the information from the certificate into the first field and then copy the request into the second field. After a short time it will show you if what’s requested is really what you got.
If, on the other hand, you change the extension to “.cer” you’ll be able to open it with Crypto Shell Extensions (which seems to be built into Windows 7+). Click the Details tab and it’s possible to see the date that the certificate is “Valid to.” And yes, if the time runs out the certificate will suddenly stop working, so keep a list of certificate expiration dates and make sure you put them on your calendar.
When you look at the certificate with a .cer extension, you can also check the serial number of the certificate, which might come in handy if you feel that a certificate of some kind on your server is wrong; like maybe it’s an old version that didn’t really get replaced in a replication when you put your shiny new certificates on the Publisher (been there).
Also, click the Certification Path tab and it will be possible to see the certificate chain involved in the generation of this bad-boy. There’s always going to be a root certificate. It’s often the case that there will be one or more intermediary certificates too. If there are several certificates listed in the path, it will be necessary to get all of them from the CA before beginning the process of loading anything onto the Cisco servers. You have to install the certificates in the order in which they are listed in the Certification Path beginning with the root and moving down the chain to the certificate generated from the CSR.
When you’ve gathered your certificates together it will be time to upload them. Go to the OS Administration page on the Publisher and navigate to Security > Certificate Management. Click the button to “Upload Certificate/Certificate Chain.” Search for the root certificate supplied by the CA and upload it as a “tomcat-trust.”
If there are any intermediary certificates, upload them as “tomcat-trust” certificates in the same order as they were displayed in the Certification Path tab on the certificate itself.
When this is done it’s a good idea to check a sampling of your cluster’s Subscribers to make sure that the root and intermediary certificates replicated like they’re supposed to.
Finally, upload the certificate generated from the CSR. I usually do it with the certificate carrying a “.pem” extension, but a “.cer” works too. It all winds up performing the same function. When you upload this multi-server certificate, though, make sure you do it as a “tomcat” certificate.
Note what I just said there. The other certificates are “tomcat-trust,” but this one is “tomcat” only.
Once the certificates are uploaded, restart the Cisco Tomcat service on each server. That’s each server. In the CLI, type “utils service restart Cisco Tomcat.” Remember, too, that when you issue this statement it usually takes a minute or so for the service to restart, but it takes about 10 minutes for the service to want to begin displaying the web page again, so make sure you do this during a maintenance window.
That should do it for the Tomcat piece. When your administrative or user pages are displayed they should now no longer show the scary message about the website being unsecure. You should also see a friendly looking lock in the address bar of the web page.
Network Stability Through Resilience Engineering
Cloud Security 101
BGP Traffic Engineering
Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.
Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.
John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services. Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.
He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.