To Make A Server Crash, You Must Find The Right Tool For The Job
   

This is going to be a short post. I have been dealing with a few server crashes recently. Some of them having to do with Tivoli and others … well, the PMRs are still opened. However, I got sick and tired of always manually collecting the IBM_TECHNICAL_SUPPORT folder, the log.nsf, the notes.ini, the Event Viewer Files and the WinMSD NFO file each time I was opening a PMR for a crashed server.
   

So, I wrote myself a nice little batch file that does all that for me. Once I had the batch file done, I decided to try it out in the “Run this Script After Server Fault” field of one of the servers in the test environment. But, to test it out, I needed to “create” a server fault.
  

And here’s the gist of this post: for those that don’t know it, there is a small utility on the Lotus Developer Domain Sandbox on a page titled “Utilities to crash client and server”. You can download it from here. The page might say that the platform is “AIX 64 bits” but the zip file contains every possible flavor of OS you can imagine. The version for Windows servers is at the root of the unzipped directory structure.
  

And what happens when you run it? Well, you get a nice “PANIC” error … your server crashes and NSD fires off. Simple is beautiful (in most cases).
   

So, for a guy like me testing out and debugging his “post-server-crash” script, this is very usefull … and by posting it on this blog, maybe someone else will discover it too (hopefully not to wreak havoc in his own environment).
   

Side Notes
  

Well, truth be told, my script works fine when I run it manually from the command prompt. However, when the server fires it off … nothing appears on the screen but files are being copied to the right place by the batch file and then zipped. It’s all done silently.
   

The catch 22 however is that I wanted my script to delete the log.nsf and run a fixup -q on the usual databases that go bonkers after a server fault (admin4.nsf, events4.nsf, mail.boxes, names.nsf just to name a few) so that the server would come up a bit more clean than in cases where it starts up right away and is catching up doing the fixup on the databases while it’s trying to get back up.
  

So, because of that, I opened another PMR with Lotus Support to ask if there is some sort of switch or notes.ini that I should try to make my script run after NSD has done what it’s supposed to do but *before* the server is restarted … we’ll see what support has to say. However, if you are a guru in regards to “scripts to run after a server fault”, please feel free to post in the comments section your 2 cents on why this is happening to me (and your idea for solutions if you have any … thanks!).
  

Conclusion
  

Please use the utility responsibly! Don’t use it for your next April Fools prank … seriously, don’t. Friends don’t let Friends crash their Friends servers as an April Fool’s joke …
 

Thanks for reading!
  

Marc

Summary …

Last week I upgraded a partitioned server in one of our remote office in the Asia-Pacific region, using Windows Remote Desktop, from Lotus Domino R7.0.2 to R8.5.1. Things went very smoothly during the upgrade but it took longer than I had planned. So, once the partitioned servers were upgraded, I started them and let them run because I didn’t want to bust my maintenance schedule window.

Because the upgrade took longer, I knew that I had to schedule some more downtime this week to be able to wrap things up and run a compact on the databases with the server down to upgrade them to the newest On Disk Structure (ODS). After some discussions with the IT folks over in the Asia-Pacific region, the next window of opportunity to schedule some downtime happened to be today, Friday, November 13th 2009.

Right away, something *should* have clicked in my head but I guess I’m so amazingly tired that none of my usual paranoia alarms went off. C’mon we’ve got movies such as “Friday the 13th” that clearly illustrate that it’s a bad day to do anything important (let alone server maintenance) … so something should have clicked in my head but alas … nothing … so read on for the horror story … or skip to the “How did I fix it?” part to read about what went wrong and how to fix it if you run into it.

But What Did I Really Need To Do On That Partitioned Server?

So, if you are still reading this, you may ask, “Sir, what did you really need to do on that particular server?”. And the answer is quite simple … I just needed to un-install one of the un-used partitions on the server and run a compact on some of the databases of the 2 other partitions to bring them up to the latest-and-greatest-omg-it-slices-and-dices-but-wait-theres-more ODS level (51). Simple enough right? Nothing that would scare the pants off your usual run-of-the-mill Lotus Domino Administrator as far as I know.

Ok, So … What Happened?

Wow, you’re still reading this? Thanks! Well, I “remote desktoped” into the server and un-installed the unused partition and that went well.

I also ran a compact on the databases to upgrade them to the latest ODS … and that also went well.

I then rebooted the server to complete the maintenance (I always like to do reboots just to clear up the Windows Server memory) … and I waited. And … I waited some more. And … some more. After 10 minutes of waiting without being able to remote desktop back into the server, it was clear that something was wrong. I tried to “ping” the server but it would not even respond … I thought “oh my, Windows must have Blue Screened” …

And It Got Worse … Right?

Yep, it did but not in the way that you’d expect. Long story short, I had (politely) asked for the login information for the ILO (HP’s Integrated Lights Out) to be added to the list of ILO information in a database that we have where I work. That usually covers our lower-back-part in case Windows crashes because ILO allows you to remote control the machine via another interface. I assumed, and trusted, that people would have done that already because I usually bend-over-backwards for them when they ask me something … and sadly, my assumption was horribly wrong because the ILO information wasn’t anywhere to be found!

Knowing full well that it was lunch time for me and midnight for the folks over in the Asia-Pacific region, I said “dammit (jim) this is an emergency, so I have the rights to wake one of guys over there up” … and I tried to call the cell phone of the LAN Admin whom I knew the ILO information. But … no answer! I then tried his home phone number: no answer either. I re-tried his cell. Still no answer. Plan B: I decided to punish call another LAN Admin in the Asia-Pacific region … also no answer!

At that point, I knew I was in deep doo doo! Finally, I asked the Senior LAN Admin for the Americas region of the company that I work for to try to find this info. Lucky for me, after about 30 minutes, he managed to find an old reference to it somewhere in his emails!

Phew … You Had Access To The Machine … What Was Wrong?

Once we ILO’ed back into the machine, we saw that it was stuck at “Applying Preferences”. After some more waiting, we ran out of patience and rebooted it. Too bad for us: it got stuck at the same place! After 2 more reboots for good measure and one final reboot in Safe Mode with Networking, the Senior LAN Admin for my region figured out what was wrong: the server was freezing when it was trying to start the Lotus Domino Server partitions!

So, he set them to run manually, rebooted and handed me back the control while he went back to fighting fires in the Americas region.

How did I fix it?

Once I was back into the server (again, via Remote Desktop), I went to the Windows Services panel and started one of the Lotus Domino Partitions. An error that I had previously ran into instantly reared its ugly head … and what was the error?

It was the good ol’ “An error occurred during license use management initialization. Ensure that you are running Domino with a valid license file” error. You can read about that nightmarish upgrade where I ran into this error for the first time on one of my 1st blog post here.

And the solution when you run into that nice ”An error occurred during license use management initialization …” error hasn’t changed since I last ran into it: simply re-run the Lotus Domino R8.5.1 installer and it will fix it automagically (see the IBM technote here).

So, now that everything is back up again … I promise never to do Server Maintenance on a Friday the 13th ever again …

Summary

A long time ago, someone needed to get something done in your company and the solution was to purchase some sort of product that you installed on top of Lotus Domino. Since then, you’ve been locked in the eternal 3 steps dance of “(step 1) a new version of Lotus Domino gets released but (step 2) wait for vendor to release compatible version of the add-in and (step 3) finally upgrade both products”. Point-in-case where I work: we needed a Fax solution that integrated with our Lotus Notes and Domino environment so someone installed FastFax from Quadrant Software on top of one of our Lotus Domino servers.

After a few years and some issues here and there, it’s been pretty much humming along. But the last issue I ran into left me wondering if there was a (free) way to monitor the 2 tasks that I see for FastFax when I issue a SHOW TASK command on the server console. So, I turned to DDM and Event Handlers for that task but, out-of-the-box, I realized that they can only be used to monitor Lotus Domino server tasks.

So, what did I do after I realized this? Well, I tried to hack Event Handlers and, oddly enough, it worked!  Here’s what I did …

Hacking Event Handlers to Monitor Add-Ins from other Vendors

Two Things Right away … First, I’m sorry if this has been written about somewhere else. I’m sure that in the Yellowverse, someone, somewhere did the exact same thing and blogged about it but I didn’t look hard enough to find it. And secondly, I’ll be blunt, this worked for me and the issue that I was faced with. There’s a good chance that this might not work for you … but you won’t hurt anything by trying. Truth be told, I’m not even sure why it works but it does…

Step 1 … issue a SHOW TASK and make a note of the Task(s) that you want Event Handlers to monitor. In my case, FFXGWOT and FFXGWIN:

Database Server      Platform Stats is gathering statistics
Database Server      Shutdown Monitor
Database Server      Process Monitor
FFXGWOT              Idle
FFXGWIN              FFXGWIN
Router               Dispatch: Idle
Router               Sweep: Idle
Router               Utility: Idle

So, now that you know which “Add-In” tasks we want to monitor … follow the steps (for the screenshots, I’m only using the FFXGWIN task):

Step 2: Open the Monitoring Configuration (events4.nsf) on the server where the Add-In task is running.

Step 3 - Open the view "Task Status" view under the "Event Generators" category.

Step 4 - Click on the "New Task Status Monitor" view Action button.

Step 5 - Select one of the Task (*any*), the Server where the Add-In task is running and "What" to monitor (status down).

Step 6 - Let's create an agent to perform the "Hack".

Step 7 - Give the agent a name and change the agent type.

Step 8 - Make sure that the agent is private and set to run on Selected Documents.

Step 9 - Add the code to change the field "Task" to the process we want to monitor.

Step 10 - Save the agent.

Step 11 - Close Designer.

Step 12 - Select the new task monitor you created and then run the agent on it.

Step 13 - Now, make a mental note of the Event Generator number because you'll need it later.

Step 14 - Now switch to the view "Event Handlers - By Server".

Step 15 - Now click on the button to create a new Event Handler.

Step 16 - On the Basics tab, select the Trigger to be a Custom event generator.

Step 17 - On the Event tab, click on the button to select the Event.

Step 18 - Scroll down to the "Task Status" event generators and select the one you just created (this is where you need the event number to make your life easier).

Step 19 - On the "Action" tab, select the "Mail" method and type your name in the Address field (yes, you could set it to run an agent or whatever you want ... I wanted to get an email).

Step 20 - Save your new Event Handler.

Step 21 - You can see your new Event Generator under the "All Servers" category.

Addendum

So, with this Event Generator and Event Handler in place, when the task FFXGWIN goes down in a ball of flames (for one reason or another), I get a nice email. Of course, I had to repeat these steps for the FFXGWOT tasks and it also works like a charm for that task.

Like I said previously, this worked for me in my environment. Maybe it won’t work for you but it won’t cost you a dime to try … Enjoy!

Summary

Another late night programming session … your vision is getting blurry and you ran out of Red Bull a few hours before that. Somewhere in your LotusScript code, there’s 1 line with a call to the “EndSection()” method of the NotesRichTextItem class … but for one reason or another you didn’t call the “BeginSection()” beforehand (you thought you did but it’s late) … it dosen’t matter right?

So you test your agent on the server and it Panics and Faults right away! The code dosen’t even go into your ErrorHandler routine (you have one right?) … sooooo what gives?

Steps to Reproduce the Error

If you want to reproduce the error, simply create a new scheduled agent in a database on one of your Lotus Domino R8.5 Fix Pack 1 test servers (or on a production server if you enjoy the occasional lynch mob running after you around the office with torches and pitchforks – hey they say running is good for you) and paste this code into the Initialize section of the agent:

On Error Goto ErrorHandler
 
 Dim Session As New NotesSession
 Dim NewEmail As NotesDocument
 Dim NewBody As NotesRichTextItem
 
 Set NewEmail = Session.CurrentDatabase.CreateDocument
 Set NewBody = New NotesRichTextItem ( NewEmail , “Body” )
 
 Call NewBody.EndSection()
 
 Exit Sub
 
ErrorHandler:
 
 Print “An error occured in the agent MCXTestAgent”
 Exit Sub

Notice that there isn’t any call to the “BeginSection” method? Now, either let the agent run on it’s schedule and watch the server Fault OR issue a TELL AMGR RUN “YourDatabaseName.nsf” ‘YourAgentName’ command … and watch it Fault.

The Solution

While this is technically a problem with LotusScript and it should have gone into the ErrorHandler routine … it’s also, technically speaking, a problem with your code … i.e. you should have have called a “BeginSection” a couple of lines above somewhere in there. So just add the “BeginSection” call where it needs to be and enjoy.

To be safe, I opened a ticket with Lotus Support to report this “behavior”. The support rep who called me back said he was able to reproduce the error quite easily and that he opened SPR #JSHN7WBRPM in regards to this issue.

After upgrading to Notes 8.5, I noticed that when I tried importing a Lotus 1-2-3 file, I was missing the option to import Lotus 1-2-3 files into a notes view.

After some searching found this article:
The option is available in 8.0.2 but no longer in Notes 8.5. After more researching, found the following. Obviously the installation / upgrade has changed the VIEWIMP variables in the client’s notes.ini.
VIEWIMP1=Structured Text,0,_ISTR,,.CGN,.LTR,.STR,._UNKNOWN,,,1,
VIEWIMP2=Tabular Text,0,_ITAB,,.PRN,.RPT,.TAB,.TXT,.TSV,,1,
VIEWIMP3=vCard,0,_IVCRD,,.VCF,,1,
VIEWIMP4=Calendar File (.ics),0,_IICAL,,.ICS,,1,
VIEWIMP5=Comma Separated Value,0,_ICSV,,.CSV,,1,
There is no VIEWIMPn= for Lotus 1-2-3 files.
Copy the missing line from below:
VIEWIMP6=Lotus 1-2-3,0,_IWKSV,,.123,.WK1,.WK3,.WK4,.WKS,.WR1,.WRK,,4,
Restart the Notes client.

Summary

You upgrade your R7.02 FP3 server to R8.5 FP1 … before the upgrade, your iNotes clients had presence awareness enabled and it worked. Now it doesn’t work anymore. What gives? It’s actually super easy to fix …

The Solution

The solution is actually in the R85 Administrator Help database in the document titled “Setting up Lotus iNotes with Sametime”. Since presence awareness *used to work* before, you can skip pretty much everything in that help document BUT you need to check, just as it’s written in that help file, if there’s a servlets.properties file in your Domino server Data directory.

If there isn’t a servlets.properties file in your Data directory, you need to create it (it’s notepad time!) and put the following line in it:

servlet.DWABuddyList.code=com.lotus.dwa.stbuddy.DWABuddyList

Once that file is saved, simply issue a RESTART TASK HTTP command and … presto! It works again.

Easy isn’t it? Please kill me now …

Summary

You noticed a while ago that your Database Catalog (catalog.nsf) on your servers wasn’t replicating quite right. You looked into the issue and discovered that each night, the server(s) would reset back the ACL so that the LocalDomainServers group only has reader access.

This issue is actually documented in Technote #1098133 by Lotus Support. Here’s the Link: http://www-01.ibm.com/support/docview.wss?uid=swg21098133

Solution from the Technote

Well, if you look at the Technote, it says that you should create another group and give that group enough rights to be able to replicate the changes and deletions to the catalog.nsf around. Ok, I tried that but it didn’t work: the servers can’t even pull the ACL changes themselves since they only have Reader Access due to the LocalDominServers group that’s already in there. Sure, you could try to push it yourself from your own client manually … but if you have a ton of servers … it might take forever.

Solution (Plan B actually)

Plan B in this case is to add each server in your domain to the ACL of the catalog.nsf database explicitely. That gets around the groups and fixes your catalog.nsf database replication issue. But adding all your servers manually to the ACL could be a daunting task (especially if you have a ton of servers). So what do you do to make sure you don’t forget any? You write an agent! So I wrote an agent that you can use from the Domino Directory to add all your servers to the ACL of the catalog.

To use the agent, open your Domino Directory and click on the Create -> Agent menu. Give the agent a name, maybe something like “Tweak ACL of Catalog.nsf” and make sure it’s set to run on selected documents. You can also, if you are paranoid make it “private” so that nobody else can accidently call that agent from the Actions menu.

Once you’re ready, set the agent to be in LotusScript, download the TweakACLofCatalog.lss file and import it into the agent (go to the Initialize section and click on File -> Import).

Once the code has been imported into the agent, simply save it (File -> Save).

To use the agent, simply go into your Domino Directory and then open the Servers view. Once you are in the Servers view, select all your server documents and click on Actions -> Tweak ACL of Catalog.nsf (or whatever you named the agent as). That will loop in the server documents and add each selected server explicitely to the ACL of the catalog.nsf database.

Addendum

What you need to keep in mind is that this bug has been fixed in Lotus Domino 8.5 … so while you have R7 and R8.0 servers in your infrastructure, you’ll need to let the ACL of the catalog.nsf as-is. Once all your servers are upgraded, just remember to clean up the ACL of the catalog.nsf and to simply give LocalDomainServers Manager access with Delete rights and everything’s gonna be fine from that point forward.

 Summary  

You stay up late to upgrade one of your Lotus Domino Servers from R7.0.2 FP3 to R8.5 FP1 … everything went fine for the previous 4 servers so what could go wrong with this one? Well, the installer works like a charm but when you start the server, you get this error:

“An error occurred during license use management initialization. Ensure that you are running Domino with a valid license file”.

Nice error ... I had never seen that one before ...

The Solution …

Re-run the Lotus Domino 8.5 installer … but don’t take my word for it … read this Technote: http://www-01.ibm.com/support/docview.wss?uid=swg21282346

Summary …

This post is about getting the the following errors right after doing a Lotus Domino 7.0.2 FP3 upgrade to Lotus Domino 8.5. You can read the sad sad sad story below or simply skip to the solution at the end (and read the addendum if you want to cry).

FAULT REPORT: SERVERNAME/ORG (Release 8.5FP1 June 15, 2009) process nHTTP faulted at 08/08/2009 08:23:00 AM with ‘ACCESS_VIOLATION’

FAULT REPORT: SERVERNAME/ORG (Release 8.5FP1 June 15, 2009) process nHTTP faulted at 08/08/2009 08:23:00 AM with ‘ACCESS_VIOLATION’

The Sad Sad Sad Story … (of cumulative epic failures) …

So … you’ve done your homework and configured a test environment that’s like your production environment. You’ve performed a test upgrade from your current version to Domino 8.5 with the Fix Pack 1. And the results? The upgrade of your test environment upgrade went super smoothly. Maybe too smoothly …

So, you confidently schedule some downtime and wake up early in the morning on a Saturday to upgrade your main hub. After running the Lotus Domino 8.5 installer and clicking Next a couple times (after double-checking that the right paths are detected and that you are installing the right type of server) … the installation goes super smoothly. You also install the Lotus Domino 8.5 Fix Pack 1 on top just like in your test environment and once again everything goes super smoothly. Remember I wrote “maybe too smoothly above” … read on …

Confidence runs high as you start your upgraded server for the 1st time … Services -> Lotus Domino (DLotusDominoData) -> Start …

Whoohoo … the console appears! You start to breathe again! And as usual, since this is your 1st server and the domain hub, you get asked to upgrade the domino directory and everyting goes smoothly.

The minutes tick by as you look at your server upgrading the design of the system databases (ddm, mail.boxes, etc, etc).

Then … all hell breaks loose: the server faults! NSD runs! Panic ensues!

Once the server starts back up, you get an email … the fault message is this:

FAULT REPORT: SERVERNAME/ORG (Release 8.5FP1 June 15, 2009) process nHTTP faulted at 08/08/2009 08:23:00 AM with ‘ACCESS_VIOLATION’

You wonder what the h*ck went wrong … but the server comes back online, then it runs fixup on a couple of opened databases (that server isn’t running Transaction Logging) and finaly, it goes back to doing the usual post-upgrade tasks …

But then it faults again! With the same error too! So you try something: you edit the notes.ini and remove http from the ServerTasks= line just for the fun of it and start the server again.

Unfortunately, the server faults again … but this time the message is different:

FAULT REPORT: SERVERNAME/ORG (Release 8.5FP1 June 15, 2009) process nrunjava faulted at 08/08/2009 08:33:17 AM with ‘ACCESS_VIOLATION’

So … nothing makes sense anymore. You’ve never seen this error before (in 12 years). For the fun of it, you Google the error … nothing “good” really comes up. Sure, a few hits and there but nothing that’s a 100% match to this issue. So, you try to check on the IBM Support web site, in the forums, on planetlotus.org and even on some of your favorite bloggers web sites … nothing … no solution to this weird problem.

But wait! You’ve got a support contract right? So, you open a SEVERITY 1 ticket with Lotus Support via the ESR web site … which turns out to be an epic failure: the system says that they can’t give you your PMR number yet because the system is being serviced/under maintenance or some sad excuse (the server is on fire maybe?).  Unphased by the epic failure of the Lotus Support ESR web site … you call up the 1-800 number to get some help. After a few minutes of punching keys on the phone to get to Lotus Support you get to speak to a rep that willl open the PMR. So, you speak with “Carol” (who happens to speak good English), she writes down your information, the topic for your ticket and the fact that it’s a SEVERITY 1 ticket (important fact). You start to see a glimmer of hope. But she she wait until the end of the call to crush that hope by saying that she can’t give you a number either because their system down. So, two epic failures in a row … it’s looking quite bleak at that point in time.

So, you politely ask Carol to escalate this to her manager and to find a way to get some help because this is a SEVERITY 1 ticket and you need help *now* … not in 4-5 hours like a Severity 3 or 4 ticket (it’s gotten that bad now that they are on call-back only). After what appears to be 30 minutes … you haven’t gotten a call back yet. So, you do what every good person-in-a-hurry does: you call back Lotus Support and, again, you speak with Carol who tries her best to act suprised that you haven’t gotten a call back yet.  After another discussion with Carol … you finaly get a call back from a nice lady named “Regina Muller” … of Passport Advantage! What-the-h*ck? So you explain to Regina that you have no idea how she got involved into this but that she can’t help me with this fault issue. So … back on the line with Lotus Support to speak again with Carol. After what appears to be another 30 minutes … you get a call back from Mr. Patrick Rowan … the person who appears to be in charge of the Lotus Support helpdesk during that weekend. Phew … hope who had completly fled the scene of the accident starts to peek it’s head around the corner … it’s kinda hoping it’s not going to get hit again … but it never bothers to look behind him for the 18 wheeler barreling down at full speed!

During the conversation, Mr. Patrick Rowan proceeds to explain that he can’t give you a PMR number yet and that since they can’t give you a PMR number, they can’t ask anybody to call you back! What a sad excuse coming from a big company. So you explain to Mr. Rowan that this is a SEVERITY 1 problem, that you are down and since this is your main hub, ldap isn’t running and it’s affecting other systems. Many other systems.

After what appears to be 1 more hour, Mr. Rowan calls you back …. hooray! He’s got a PMR number for you … ##### branch 999 … in 12 years of working with Lotus Domino, this is the first time ever in your short life that you’re given a PMR number with 999 as the “branch” instead of 005. Deep down inside, you feel this is going to end badly. Anyhow, you decide to ignore that little voice inside screaming because finaly … it feels like you’re going get some help (and you’ve been on-and-off with Lotus Support for 2 hours so far).

Finaly, at one point you get a call back from a gentlemen named Mr. Dale Cole … and after sending him various NSD files and trying a couple different things on the server … he can’t help you. So, Mr. Cole wakes up pulls another support rep named “Anur” (but you’re not sure about his name due to the accent and he never mentions his last name saddly). Both of them ask you to do various unholy things to your server … and nothing works. You are contemplating giving up and handing in your resignation letter (which you conviently updated a few days before and stored it on your network drive for such an occasion). You also consider declaring a disaster … but at this point you figure out that with the combined brain power of 3 people on this conference call … someone will eventually find out what the h*ck is wrong with the server.

Saddly that’s not the case … after trying so many different things … nothing works … so … the only solution left is quite simple (and sad by my standards).

The Solution … (the sad sad sad solution) …

After a lot of discussions with Dale and Amur … the only solution left is to:

(a) backup the notes.ini and server id file.

(b) Uninstall the Lotus Domino server.

(c) Rename the left-over Lotus\Domino folder to Lotus\Domino-Old .

(d) Install Lotus Domino 8.5 and then install the Lotus Domino 8.5 Fix-Pack 1.

(e) Copy back the notes.ini into the newly-recreated C:\Lotus\Domino.

(f) Edit the notes.ini to make sure that the “servicename=” line matches what’s in the registry.

(g) Start the server.

After this “clean” install … the server runs like a charm … no faults and no errors. Oh, I almost forgot to mention: http and runjava are also running fine.

Truth be told, Lotus Support was never able to tell me what went wrong or which file was “left-over” in the C:\Lotus\Domino\JVM folder and not overwritten by the Lotus Domino 8.5 Installer. I took a look inside the C:\Lotus\Domino-Old folder that I sill had on the hard drive but there were too many jar files left over … so which one was the cause? I have no clue. Why didn’t the 8.5 installer didn’t remove them or didn’t overwrite them is beyond me.
 

So … I hope this post helps someone somewhere.

Addendum

Wanna laugh? Or cry maybe? Remember the ticket that I *tried* to open via the ESR web site that was “in limbo” and that the system told me it couldn’t assign a number to it? Well, on Monday night … a full 2+ days after … I got a call from a lady (I didn’t get her name) from Lotus Support … she was asking me if my problem with our Lotus Domino 8.5 upgrade on our ZSeries server was fixed now! I explained to her that we don’t have any mainframe (ZSeries) and that I had opened a Sev 1 ticket for a Lotus Domino 8.5 upgrade issue on a Windows platform … and that a call back 48 hours later isn’t really good timing in my books. Oh well … it made me laugh.