I’m Serious: Seamless Mail Upgrade Needs a Progress Bar …
  

Lesson Learned: I just did an upgrade from Release R7 to R8.5.1 of a “remote site”, i.e. a location that goes across the WAN to get to their Notes server … and Seamless Mail Upgrade gave me some trouble when it comes to the “folder upgrade” part.
 

Basically speaking, if you use the “Seamless Mail Upgrade” to upgrade those user’s mail files via policy, they will get prompted to upgrade their folders after they open their mail file for the 1st time. 
 

But the problem is as follows: if you have a user with a ton of folders … it will take forever to upgrade their folders across the WAN (I’ve seen it take 3 hours on 1 particular machine) … and there is NO PROGRESS BAR. Here’s an example of what their screen looks like … interesting isn’t it? Apart from the usual “lightning bolt” in the lower left part of the screen, there is nothing that tells the user what’s going on.
     

What do most users think? They Think It FROZE!
 

Like I said, these were users going across the WAN and their “Seamless Mail folder” upgrade took a while (hours for some). Some of those employees thought that their Lotus Notes client had frozen and rebooted their machine … with the effect that they got re-prompted to upgrade their folders again (because the notes.ini variable “MailUpgradeFolder” is set to the same value as the “Setup” variable in the notes.ini once the Mail Folder upgrade is completed … and since they stopped it, it isn’t set).
  

And What’s The Solution? Communication … and an SPR
 

If you really want to use the Seamless Mail Upgrade, then the only thing you can do for now is to tell your employees that are in remote sites that the Seamless Mail Upgrade might take a while … a long while in fact.
  

If you want to turn it off, you can. But then you will have to do the mail template upgrades yourself at night (or during the day) and I’m pretty sure that you have better things to do than upgrade mail templates at night.
 

And last but not least: the SPR! Shortly after running into this, I opened a ticket with Lotus Support to request an SPR to add a progress bar to the Seamless Mail Template upgrade. The SPR number is RSSA87HKV7 and feel free to open a ticket with Lotus Support to add your name to the list of customers who need the same thing … the more people ask, the more chances of this making it to the next release.
  

Addendum…

It’s been an amazingly busy 2 months for me and I haven’t had the time to blog a lot. I hope that things will quiet down a bit in August so that I can put the finishing touches of the 4th version of the Smart Upgrade guide (I need to add a section about the Widget catalog and the Update Site).
 

Thanks for reading!
 

Marc

After upgrading to Notes 8.5, I noticed that when I tried importing a Lotus 1-2-3 file, I was missing the option to import Lotus 1-2-3 files into a notes view.

After some searching found this article:
The option is available in 8.0.2 but no longer in Notes 8.5. After more researching, found the following. Obviously the installation / upgrade has changed the VIEWIMP variables in the client’s notes.ini.
VIEWIMP1=Structured Text,0,_ISTR,,.CGN,.LTR,.STR,._UNKNOWN,,,1,
VIEWIMP2=Tabular Text,0,_ITAB,,.PRN,.RPT,.TAB,.TXT,.TSV,,1,
VIEWIMP3=vCard,0,_IVCRD,,.VCF,,1,
VIEWIMP4=Calendar File (.ics),0,_IICAL,,.ICS,,1,
VIEWIMP5=Comma Separated Value,0,_ICSV,,.CSV,,1,
There is no VIEWIMPn= for Lotus 1-2-3 files.
Copy the missing line from below:
VIEWIMP6=Lotus 1-2-3,0,_IWKSV,,.123,.WK1,.WK3,.WK4,.WKS,.WR1,.WRK,,4,
Restart the Notes client.

Summary

You upgrade your R7.02 FP3 server to R8.5 FP1 … before the upgrade, your iNotes clients had presence awareness enabled and it worked. Now it doesn’t work anymore. What gives? It’s actually super easy to fix …

The Solution

The solution is actually in the R85 Administrator Help database in the document titled “Setting up Lotus iNotes with Sametime”. Since presence awareness *used to work* before, you can skip pretty much everything in that help document BUT you need to check, just as it’s written in that help file, if there’s a servlets.properties file in your Domino server Data directory.

If there isn’t a servlets.properties file in your Data directory, you need to create it (it’s notepad time!) and put the following line in it:

servlet.DWABuddyList.code=com.lotus.dwa.stbuddy.DWABuddyList

Once that file is saved, simply issue a RESTART TASK HTTP command and … presto! It works again.

Easy isn’t it? Please kill me now …

Summary

You noticed a while ago that your Database Catalog (catalog.nsf) on your servers wasn’t replicating quite right. You looked into the issue and discovered that each night, the server(s) would reset back the ACL so that the LocalDomainServers group only has reader access.

This issue is actually documented in Technote #1098133 by Lotus Support. Here’s the Link: http://www-01.ibm.com/support/docview.wss?uid=swg21098133

Solution from the Technote

Well, if you look at the Technote, it says that you should create another group and give that group enough rights to be able to replicate the changes and deletions to the catalog.nsf around. Ok, I tried that but it didn’t work: the servers can’t even pull the ACL changes themselves since they only have Reader Access due to the LocalDominServers group that’s already in there. Sure, you could try to push it yourself from your own client manually … but if you have a ton of servers … it might take forever.

Solution (Plan B actually)

Plan B in this case is to add each server in your domain to the ACL of the catalog.nsf database explicitely. That gets around the groups and fixes your catalog.nsf database replication issue. But adding all your servers manually to the ACL could be a daunting task (especially if you have a ton of servers). So what do you do to make sure you don’t forget any? You write an agent! So I wrote an agent that you can use from the Domino Directory to add all your servers to the ACL of the catalog.

To use the agent, open your Domino Directory and click on the Create -> Agent menu. Give the agent a name, maybe something like “Tweak ACL of Catalog.nsf” and make sure it’s set to run on selected documents. You can also, if you are paranoid make it “private” so that nobody else can accidently call that agent from the Actions menu.

Once you’re ready, set the agent to be in LotusScript, download the TweakACLofCatalog.lss file and import it into the agent (go to the Initialize section and click on File -> Import).

Once the code has been imported into the agent, simply save it (File -> Save).

To use the agent, simply go into your Domino Directory and then open the Servers view. Once you are in the Servers view, select all your server documents and click on Actions -> Tweak ACL of Catalog.nsf (or whatever you named the agent as). That will loop in the server documents and add each selected server explicitely to the ACL of the catalog.nsf database.

Addendum

What you need to keep in mind is that this bug has been fixed in Lotus Domino 8.5 … so while you have R7 and R8.0 servers in your infrastructure, you’ll need to let the ACL of the catalog.nsf as-is. Once all your servers are upgraded, just remember to clean up the ACL of the catalog.nsf and to simply give LocalDomainServers Manager access with Delete rights and everything’s gonna be fine from that point forward.

 Summary  

You stay up late to upgrade one of your Lotus Domino Servers from R7.0.2 FP3 to R8.5 FP1 … everything went fine for the previous 4 servers so what could go wrong with this one? Well, the installer works like a charm but when you start the server, you get this error:

“An error occurred during license use management initialization. Ensure that you are running Domino with a valid license file”.

Nice error ... I had never seen that one before ...

The Solution …

Re-run the Lotus Domino 8.5 installer … but don’t take my word for it … read this Technote: http://www-01.ibm.com/support/docview.wss?uid=swg21282346

Summary …

This post is about getting the the following errors right after doing a Lotus Domino 7.0.2 FP3 upgrade to Lotus Domino 8.5. You can read the sad sad sad story below or simply skip to the solution at the end (and read the addendum if you want to cry).

FAULT REPORT: SERVERNAME/ORG (Release 8.5FP1 June 15, 2009) process nHTTP faulted at 08/08/2009 08:23:00 AM with ‘ACCESS_VIOLATION’

FAULT REPORT: SERVERNAME/ORG (Release 8.5FP1 June 15, 2009) process nHTTP faulted at 08/08/2009 08:23:00 AM with ‘ACCESS_VIOLATION’

The Sad Sad Sad Story … (of cumulative epic failures) …

So … you’ve done your homework and configured a test environment that’s like your production environment. You’ve performed a test upgrade from your current version to Domino 8.5 with the Fix Pack 1. And the results? The upgrade of your test environment upgrade went super smoothly. Maybe too smoothly …

So, you confidently schedule some downtime and wake up early in the morning on a Saturday to upgrade your main hub. After running the Lotus Domino 8.5 installer and clicking Next a couple times (after double-checking that the right paths are detected and that you are installing the right type of server) … the installation goes super smoothly. You also install the Lotus Domino 8.5 Fix Pack 1 on top just like in your test environment and once again everything goes super smoothly. Remember I wrote “maybe too smoothly above” … read on …

Confidence runs high as you start your upgraded server for the 1st time … Services -> Lotus Domino (DLotusDominoData) -> Start …

Whoohoo … the console appears! You start to breathe again! And as usual, since this is your 1st server and the domain hub, you get asked to upgrade the domino directory and everyting goes smoothly.

The minutes tick by as you look at your server upgrading the design of the system databases (ddm, mail.boxes, etc, etc).

Then … all hell breaks loose: the server faults! NSD runs! Panic ensues!

Once the server starts back up, you get an email … the fault message is this:

FAULT REPORT: SERVERNAME/ORG (Release 8.5FP1 June 15, 2009) process nHTTP faulted at 08/08/2009 08:23:00 AM with ‘ACCESS_VIOLATION’

You wonder what the h*ck went wrong … but the server comes back online, then it runs fixup on a couple of opened databases (that server isn’t running Transaction Logging) and finaly, it goes back to doing the usual post-upgrade tasks …

But then it faults again! With the same error too! So you try something: you edit the notes.ini and remove http from the ServerTasks= line just for the fun of it and start the server again.

Unfortunately, the server faults again … but this time the message is different:

FAULT REPORT: SERVERNAME/ORG (Release 8.5FP1 June 15, 2009) process nrunjava faulted at 08/08/2009 08:33:17 AM with ‘ACCESS_VIOLATION’

So … nothing makes sense anymore. You’ve never seen this error before (in 12 years). For the fun of it, you Google the error … nothing “good” really comes up. Sure, a few hits and there but nothing that’s a 100% match to this issue. So, you try to check on the IBM Support web site, in the forums, on planetlotus.org and even on some of your favorite bloggers web sites … nothing … no solution to this weird problem.

But wait! You’ve got a support contract right? So, you open a SEVERITY 1 ticket with Lotus Support via the ESR web site … which turns out to be an epic failure: the system says that they can’t give you your PMR number yet because the system is being serviced/under maintenance or some sad excuse (the server is on fire maybe?).  Unphased by the epic failure of the Lotus Support ESR web site … you call up the 1-800 number to get some help. After a few minutes of punching keys on the phone to get to Lotus Support you get to speak to a rep that willl open the PMR. So, you speak with “Carol” (who happens to speak good English), she writes down your information, the topic for your ticket and the fact that it’s a SEVERITY 1 ticket (important fact). You start to see a glimmer of hope. But she she wait until the end of the call to crush that hope by saying that she can’t give you a number either because their system down. So, two epic failures in a row … it’s looking quite bleak at that point in time.

So, you politely ask Carol to escalate this to her manager and to find a way to get some help because this is a SEVERITY 1 ticket and you need help *now* … not in 4-5 hours like a Severity 3 or 4 ticket (it’s gotten that bad now that they are on call-back only). After what appears to be 30 minutes … you haven’t gotten a call back yet. So, you do what every good person-in-a-hurry does: you call back Lotus Support and, again, you speak with Carol who tries her best to act suprised that you haven’t gotten a call back yet.  After another discussion with Carol … you finaly get a call back from a nice lady named “Regina Muller” … of Passport Advantage! What-the-h*ck? So you explain to Regina that you have no idea how she got involved into this but that she can’t help me with this fault issue. So … back on the line with Lotus Support to speak again with Carol. After what appears to be another 30 minutes … you get a call back from Mr. Patrick Rowan … the person who appears to be in charge of the Lotus Support helpdesk during that weekend. Phew … hope who had completly fled the scene of the accident starts to peek it’s head around the corner … it’s kinda hoping it’s not going to get hit again … but it never bothers to look behind him for the 18 wheeler barreling down at full speed!

During the conversation, Mr. Patrick Rowan proceeds to explain that he can’t give you a PMR number yet and that since they can’t give you a PMR number, they can’t ask anybody to call you back! What a sad excuse coming from a big company. So you explain to Mr. Rowan that this is a SEVERITY 1 problem, that you are down and since this is your main hub, ldap isn’t running and it’s affecting other systems. Many other systems.

After what appears to be 1 more hour, Mr. Rowan calls you back …. hooray! He’s got a PMR number for you … ##### branch 999 … in 12 years of working with Lotus Domino, this is the first time ever in your short life that you’re given a PMR number with 999 as the “branch” instead of 005. Deep down inside, you feel this is going to end badly. Anyhow, you decide to ignore that little voice inside screaming because finaly … it feels like you’re going get some help (and you’ve been on-and-off with Lotus Support for 2 hours so far).

Finaly, at one point you get a call back from a gentlemen named Mr. Dale Cole … and after sending him various NSD files and trying a couple different things on the server … he can’t help you. So, Mr. Cole wakes up pulls another support rep named “Anur” (but you’re not sure about his name due to the accent and he never mentions his last name saddly). Both of them ask you to do various unholy things to your server … and nothing works. You are contemplating giving up and handing in your resignation letter (which you conviently updated a few days before and stored it on your network drive for such an occasion). You also consider declaring a disaster … but at this point you figure out that with the combined brain power of 3 people on this conference call … someone will eventually find out what the h*ck is wrong with the server.

Saddly that’s not the case … after trying so many different things … nothing works … so … the only solution left is quite simple (and sad by my standards).

The Solution … (the sad sad sad solution) …

After a lot of discussions with Dale and Amur … the only solution left is to:

(a) backup the notes.ini and server id file.

(b) Uninstall the Lotus Domino server.

(c) Rename the left-over Lotus\Domino folder to Lotus\Domino-Old .

(d) Install Lotus Domino 8.5 and then install the Lotus Domino 8.5 Fix-Pack 1.

(e) Copy back the notes.ini into the newly-recreated C:\Lotus\Domino.

(f) Edit the notes.ini to make sure that the “servicename=” line matches what’s in the registry.

(g) Start the server.

After this “clean” install … the server runs like a charm … no faults and no errors. Oh, I almost forgot to mention: http and runjava are also running fine.

Truth be told, Lotus Support was never able to tell me what went wrong or which file was “left-over” in the C:\Lotus\Domino\JVM folder and not overwritten by the Lotus Domino 8.5 Installer. I took a look inside the C:\Lotus\Domino-Old folder that I sill had on the hard drive but there were too many jar files left over … so which one was the cause? I have no clue. Why didn’t the 8.5 installer didn’t remove them or didn’t overwrite them is beyond me.
 

So … I hope this post helps someone somewhere.

Addendum

Wanna laugh? Or cry maybe? Remember the ticket that I *tried* to open via the ESR web site that was “in limbo” and that the system told me it couldn’t assign a number to it? Well, on Monday night … a full 2+ days after … I got a call from a lady (I didn’t get her name) from Lotus Support … she was asking me if my problem with our Lotus Domino 8.5 upgrade on our ZSeries server was fixed now! I explained to her that we don’t have any mainframe (ZSeries) and that I had opened a Sev 1 ticket for a Lotus Domino 8.5 upgrade issue on a Windows platform … and that a call back 48 hours later isn’t really good timing in my books. Oh well … it made me laugh.