New Nexus 9K Items
Have you ever created a new Unity subscriber and then found that the user is unable to receive voicemail? I’ve run into this several times and while there are some documented troubleshooting steps to be found, I thought I’d take the chance to run thru things in a bit more detail and actually delve into why and how you would use the Message Store Configuration Wizard to resynchronize all of the objects in Unity with the mailstore.
Issue: You create a new Unity subscriber but the subscriber cannot receive voicemail. When calling the new Subscriber, callers hear the greeting but then receive a system error message stating that a system error has occurred and to report this to the system administrator.
From a system level, an error would be logged in the Application Event Log. Specifically, the details will notify you that the SMTP Address field for the Subscriber is NULL. What this is telling you is that the SMTP address is not being written to the SQL database.
Considerations: There are actually 2 “flavors” of this issue that I’ve come across. The troubleshooting steps (and possibly root cause) may be different based on what is occurring on your system.
1. The SMTP Address is not populated in the Active Directory user object for the Subscriber (therefore, it won’t be in the Unity SQL database).
2. The SMTP address is in Active Directory; however, it is not in the Unity SQL database.
It’s very easy to verify which “flavor” of this issue is affecting your system.
1. Verify if the SMTP is populated in Active Directory.
a. On the Unity server, open Start > Programs > Microsoft Exchange > Active Directory Users and Computers.
b. Highlight the applicable Domain.
c. From the Action Menu, select Find. Then search for and locate the user.
d. Double-click the user object. If the E-mail attribute on the general properties page is not populated, you are likely running into “Flavor” 1.
If the E-mail address/SMTP address is not being updated in Active Directory, this is typically due to an issue with the Recipient Update Service (RUS) on the Exchange server. If you manage the AD/Exchange environment for Unity then the best 1st line of troubleshooting is to restart this service on the Exchange server. You restart the RUS service from within the Exchange System Manager. If you do not manage the AD/Exchange environment, you may want to work with them to resolve this issue. Once the issue is resolved on the AD/Exchange side, Unity should receive the update for the SMTP address and write the data to the SQL table on its own. You can verify this using the steps below (#2) to query the Unity database.
2. If the E-mail attribute for the user is populated (i.e., in the example above it would display DHailey@lab01.local), then you can verify the issue on the Unity side by performing the following steps.
a. On the Unity server, open Tools Depot > Diagnostic Tools > Data Link Explorer (aka, CUDLE).
b. In the Table Name column, scroll down and highlight Subscriber.
c. From the View menu, open Query Builder and type the following query as follow “SELECT Alias, SMTPAddress FROM Subscriber WHERE SMTPAddress IS NULL” (but without the quotations).
d. Select Execute. For users where no SMTP address is populated, these accounts are essentially broken and will need to be manually resolved. Please note that the Installer account should always have a NULL SMTP address entry; therefore, it can be ignored. In the example below, DHailey is broken.
Since “Flavor 1” is typically attributed to an issue within AD/Exchange, I’m going to focus more on how to resolve “Flavor 2”. Again, this version of the issue indicates that AD/Exchange is working properly; however, Unity is still out of sync and not writing SMTP addresses to the SQL database. There are many reasons why this could happen. All of the Cisco documentation I’ve seen still references AD/Exchange as primary suspects; however, I believe this could also be an issue local to the Unity server. There are a number of services involved on the Unity side that help monitor the directory, identify changes, write changes to the DB, and so forth. In the interest of time, let’s move on to how you fix this. There are two high-level tasks that you must complete:
1. Manually update the SQL database with the SMTP address for each affected user.
2. Force Unity to sync with the mailstore/directory using the Message Store Configuration Wizard.
Now, let’s get things working again:
1. Open Start > Programs > Microsoft SQL Server > Enterprise Manager.
2. Expand down to the local servers and then select Databases > UnityDb.
3. Find Tables > Subscriber.
4. Right-click the Subscriber table and you can either return all rows or build a query. I recommend performing the same query as we did with CUDLE earlier.
5. For each Subscriber with a NULL SMTP Address, you’ll need to manually fill in the primary SMTP address based on the AD object for that user (i.e., E-mail address).
6. Once you are done, click the Exclamation Point (!) or Run button in SQL.
OK – so the next step is, IMO or from what I’ve been able to find, not well documented. The first time I had to do it, I had that “what the heck is this going to do” feeling. However, it’s not so bad. Here’s where you manually force Unity to sync using the Message Store Configuration wizard. To do this, you must be logged in as a user with rights to the mailstore – personally, I use the UnityInstall account for this task.
7. Open a command line prompt and navigate to the Commserver folder on your server. In my example, I’ve loaded Unity on the D: drive so we’ll be working from that root. So, type D: and enter.
8. Type cd commserverconfigurationsetup and enter.
9. Now type setup.exe sync.
10. The Cisco Unity Message Store Configuration Wizard launches full-screen. Click Next.
11. A process runs and provides a status icon. In short, it says to please wait while Cisco Unity fully resynchronizes with the mailstore. Do not click Next. This process could take quite a long time depending on the size of the database and other factors. For reference, I did this for a customer with about 4000 subscribers and it took approximately 5 hours to complete.
12. When the sync is complete, the Wizard proceeds without prompting and indicates that it is restarting services. When done, click Finish.
If you’re interested to know what goes on in the background, you can take a look at the application log. You’ll notice that the primary directory replication and SQL change writer services stop and start. Once complete, some folks prefer to reboot the Unity server – others do not. Personally, I don’t (or have not yet). However, I have seen that it can take a while for things to completely clear up. My recommendation is to create a test subscriber, wait 15-20 minutes for AD, and then verify if the SMTP address is stamped in AD and then in SQL. If it’s not written to SQL right away, be patient and check again within a few hours…you may find it just took a while. Of course, if you still have issues after performing these procedures…it may be time for a TAC case.
Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.
Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.
John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services. Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.
He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.