My Amazing Manager (Patrick Darke) Likes To Try New Phones…

So, I was not really surprised when he showed me his brand new Galaxy Nexus… however he was only showing it to me because he could not activate it with Traveler.

Basically speaking, every time he would go to log into the Traveler server on his Galaxy Nexus phone, he would get a message on his phone that said:

Error 500

HTTP Web Server: Invalid POST Request Exception

At first, I thought something had gone wrong on the Traveler server (we are currently running R8.5.2.4)!

I checked and the server was OK *and* I could access the Traveler server web site on my own Android phone (running Gingerbread 2.3.something).

A couple of Google searches later that returned no solution to this, I did what I always do: I called up Lotus Support.

The Solution…

Well, after opening the call, I got a call back 1.5 hour later from a support rep that told me very quickly the solution.

(1) Upgrade your Lotus Traveler server to R8.5.3.1.

(2) Download from Lotus Fix Central the Interim Fix we posted yesterday (December 19th) to add Android 4.0 support.

The Interim Fix on Fix Central is titled “LNT_8531_20111207_1606_ANDROID” (see screenshot below) but the support rep sent me a link to a different one that ends with “1619″ … so I guess I got a newer version (?).

So tonight, it’s Lotus Traveler server upgrade time for me!

Conclusion?

Well, not much to say… I just hope it helps someone, somewhere out there.

Thanks for reading!

Marc

This Story Has It All…

Suspense! Wonder! Stress! And bad customer service from Dell to boot!

It’s got everything except Bond Girls and explosions… so long story short, if your Dell MD3200 Storage Array suddenly starts acting completely wonky, your VMware ESXi servers can’t see any of the LUNs on it and are getting bad customer service from Dell… you might want to try this…

… simply re-create the Host Group configuration!

Yep, that did the trick for us.

You can read the full [and sad] story on my other blog (www.thenewvmadmin.com) via this link.

Conclusion?

None really… this is just a “heads-up” in case anybody runs into something like this on an MD3200.

Thanks for reading!

Regards

I Just Had A Weird One…

While troubleshooting a DAOS issue with XLS files becoming unreadable, I noticed that could not use the “Files” tab in my Administrator client (R8.5.2 FP2) anymore to browse all the files on the server running DAOS .

My admin client would stop after loading a few databases and say “Error getting files on ARCHIVES/UU – Database list is full” in the status bar at the bottom.

Here’s a screen shot:

The Simple Solution…

Oddly enough, I was on the phone with Lotus Support about a DAOS issue when this happened… so the rep did a quick search and found Technote #1438610 titled “Administration Client does not list all files under “Files” tab” (here’s a link) and the solution is super simple:

1) Exit the Administration Client.

2) Rename the domadmin.nsf and dommon.nsf databases in the Notes Data Directory.

3) Restart the Administration Client. Those databases will be rebuilt automatically.

That’s it… that’s all! I wonder what causes it however. The Technote says it applies to 8.0.2 but I’m running 8.5.2 FP2 so I found that a bit odd.

Conclusion / In Other News…

In other news, in working with Lotus Support about my DAOS issue, the rep told me something so amazingly and incredibly dumb about DAOS that I will just have to blog about it.

I’m giving Lotus Support the “benefit of the doubt” for now and I will wait a bit before posting something about my DAOS issue and what I was told.

Thanks for reading!

Marc

I Called Lotus Support Often Over The Years…

Maybe a bit too much: I used to l-o-v-e Lotus Support! I got great service, fast response times, and generally had an excellent experience with them. I still like them… but the magic is gone.

Nowadays, service is a “hit or miss”. Response times have gone out the window. My “call back” preferences are ignored 50% of the time. And to top it off, I’ve had really bad experiences recently with some support reps.

If you wonder if I have anything to “back my claims”, here’s a graph that indicates how often I called Lotus Support over the past 10 years.

If you wonder what happened in 2010, it’s very simple: that’s the year we deployed Lotus Notes R8.5.1. It’s as simple as that: we had a few problems with it.

So, Where’s Pay-Per-Incident?

As we prepare to move away from Lotus to Google Apps (see my blog post here) we will not renew our support agreement when it comes up for renewal (soon). We won’t get rid of Lotus Notes & Domino right away… we’ve got too many apps on that platform

So my manager Patrick asked me to check with Lotus Support the cost for “Pay Per Incident” calls with Lotus Support.

I called up Lotus Support and after 1 transfer, 1 disconnect later and 1 call back, I was able to speak with a customer service rep named “Magan” who told me that Lotus Support does not offer “Pay Per Incident” anymore.

That’s odd! BlackBerry still does (don’t take my word for it: check this page on their site for details) and VMware also offers that for some of it’s product (once again, check this page and this page on their web site for the details).

So, does anybody remember “when” Lotus Support stopped offering “Pay Per Incident” (if they ever did – I’m pretty sure that “yes” they did) and “why” they stopped offering that? Leave a message in the comments if you know the answers to those questions.

Conclusion?

Well, that was just a simple blog post … nothing really to write home about: it’s a simple question that I can’t find answer for easily.

Thanks for reading!

Marc

This Will Be A Quick Blog Post…

I ran into something interesting in my VMware environment today. One of the PC LAN Admins had to reboot an ESXi 3.5 server that runs one of our Lotus Domino servers and got a nice error message in vCenter after the reboot.

Basically speaking, the PC LAN Admin and I got this error message when we tried to access that old ESXi 3.5 host it via our vSphere clients:

The license server is not configured to perform the operation

That’s an odd error to get… especially since everything has been running super smoothly in my VMware environment. Oh and if you wonder “why” we are still running ESXi 3.5 it’s because we do have a few “old” servers with 32 bit processors. Simple as that.

The Root Cause & The Solution…

Well, I won’t repeat myself but if you are interested in reading about the root cause and the solution, feel free to head over to my other blog (The New VMware Admin) and read my post titled “vCenter 4.1 Update 2: Your License Server Settings Will Be Wiped Out During Your Upgrade“.

Conclusion?

Well, lots of us now have Lotus Domino running on VMware… and it works like a charm 99.999% of the time.

So, I decided to “cross-post” this on my Domino blog just in case someone else, somewhere, runs into the same error.

Thanks for reading!

Marc

I Got A New Phone…

And I think it has anorexia! It’s a Motorola Droid RAZR from Verizon and I seriously think that this phone suffers from anorexia! It’s extremely thin and light compared to my BlackBerry Torch 9800… truth be told: I feel like I’m going to break it just by holding it! This picture might convey a bit more how “thin” that Razr is:

Does It Work Well?

Well, my previous experience(s) with Android phones weren’t that long so I don’t any “bias” toward certain Android phones or versions. In fact, I should note that I’m a BlackBerry fan… despite the fact that they can’t figure a way out of their current tailspin.

Regardless of all this, here’s a quick recap of what I like/dislike so far.

Like: I don’t have much Android experience to compare it “against” but I can tell you one thing: it’s fast!

Like: Response times to switch between apps and/or surf the web is much faster than on the previous Android phones I’ve used in the past.

Like: The screen is *huge* and easy to read. It’s not a “retina display* but it’s still very bright and colorful. Better than the screen on my BlackBerry Torch 9800 (I probably shouldn’t even compare it against that phone… the screen is horrible on my Torch).

Dislike: Battery life is horrible. Why do I say that? Well, I got that phone yesterday afternoon at 1 PM. I fully charged it and then installed Lotus Traveler, installed a few apps from the marketplace and surfed the web a bit to “test-drive” it. I did not put it on the charger overnight so this morning at 8:30 AM the battery reached 0%! I had not even placed a call yet with it!

Not Sure if I Like Or Dislike: It runs Gingerbread (Android 2.3.5)… I thought I was going to get a higher rev, 4.x aka Ice Cream Sandwich, but then again the development kit was only released a few weeks ago. Maybe we’ll get an over-the-air upgrade? I hope so anyway.

Not Sure if I Like Or Dislike: The size of the device overall. It’s bigger than my BlackBerry Torch and it’s bigger than an iPhone. Much bigger! So time will tell if I “get used” to the giant screen and device size.

To wrap this up, here’s a picture to compare side-by-side the two devices:

About Lotus Traveler…

We run Traveler 8.5.2.4 in our environment (not 8.5.3) and I was pretty sure I’d get some widgets on Android [after installing Traveler] that I could display on the device screen… that’s not the case. I will have to ask Lotus if I need to upgrade to 8.5.3 to get that or not (or if it’s just because there aren’t any widgets for my version of Android).

Also an odd coincidence: the Traveler server crashed when I did my activation for the first time… very odd timing.

Conclusion?

Well, I don’t have much to say except… “stay tuned”. I’ll probably blog in a month or two about my experience with that device.

In the meantime: thanks for reading!

Marc

This Will Be A Short Post…

Yesterday, I blogged about adding iNotes_WA_MAC_IM=1 to your servers notes.ini to enable the little Sametime chat icon in Safari on Mac’s.

After you do that, the Sametime icon appears in iNotes and you can display your contact list… but if you forget to  disable the pop-up blocker in Safari, you won’t be able to send or receive chats from anyone!

So a word of advice, remember to warn your Mac users to disable pop-up blockers if they want to be able to chat. The easiest way for a user to disable the pop-up blocker is to click on the “Safari” menu and then to remove the check-mark next to the “Block Pop-Up Windows” option:

Oddly enough #1, Safari does not display any warnings about a pop-up being blocked like FireFox does on my MacBook Pro (it displays a yellow bar at the top)… oh well.

Oddly enough #2, I could not find an option in the Safari preferences to set “exceptions” for pop-up blockers like you can in Internet Explorer or FireFox on Windows… but I’m quite new to Mac’s so I probably haven’t found “where” to set that option yet.

Conclusion?

This was just a simple blog post… something I forgot to mention in my blog post yesterday…

I hope it helps someone, somewhere…

Thanks for reading!

Marc

Mmm… That Was Weird (For Me Anyway)…

I work in a nice worldwide company that I really like. And like most companies we have standards and policies. For example, we say that we do not support any Apple computers on our network. It doesn’t mean we don’t have any Macs here and there (some graphic designers will only let you pry their Mac from their cold dead hands for example), but they are very very very rare.

So, our story begins with the fact that we have some consultants here and there in the company and we provide for them an iNotes account (not the full Notes client – just iNotes) to communicate with our employees more easily either via email or via the Sametime chat client embedded into iNotes.

Today however, I received a strange request from one of those consultants:

“Marc, why can’t I see and use the little Sametime icon when I use Safari on my MacBook Pro to access iNotes? I saw that my co-worker [another consultant with an IBM ThinkPad running Windows 7] has a little icon in his iNotes that allows him to use Sametime and chat with other people in the company… but that icon isn’t even there for me on my Mac!”

I was puzzled… so I investigated.

The Simple Solution

A few Google queries later, I stumbled upon a 2009(!) developerWorks article titled “New features in IBM Lotus iNotes 8.5: Full mode“. About 1/3 down the article there’s a paragraph that says:

Last, to use the Lotus Sametime capability on Mac browsers, the NOTES.INI setting iNotes_WA_MacIM=1 must be specified within the server’s NOTES.INI file.

Doh! I guess I missed that when I read the release notes on our last upgrade to 8.5.2…

So, after adding the line iNotes_WA_MacIM=1 to the notes.ini of the iNotes server and restarting the HTTP task (TELL HTTP RESTART) … that consultant was able to see the little chat icon in iNotes.

Conclusion?

At this point in time, I’m not sure why it’s not enabled by default… maybe it’s still not super stable and Lotus does not want this widespread? Or did I miss a new checkbox on my server documents (maybe)?.

Anyhow … I just hope this helps someone, somewhere…

Thanks for reading!

Marc

This Was Weird…

You may (or may not) have read my previous blog post in which I wrote that I was going to apply an updated Hot Fix (#NTOR-8N5C2Y) on my Sametime R8.5.2 Classic servers to, hopefully, resolve the “Timed Out” errors we’ve been seeing over the past 2 months.

I applied the Hot Fix on all 3 regional servers and each server appeared to work correctly afterwards… it turns out that there *was* a problem… just nobody was awake at that time to “see” and report it.

Something Had Gone Wrong…

This morning, I woke up to a slew of emails (no phone calls however) from employees in Asia-Pacific saying they couldn’t see any European employees online and emails from European employees saying they couldn’t see *anybody* online. That was a rough way to start my day (and I hadn’t slept much because the upstairs neighbors had been kept me awake because of the noise they made re-enacting their honeymoon night from 1 AM to 3 AM)…

So once I dragged my tired body into the office this morning, I began by asking for a call back from our Lotus Support rep (who had asked me to put the Hot Fix on our server). While waiting for his call back, I poked around the Sametime_20111123.log file on my server to see if there weren’t any “obvious” errors.

And there was some very obvious errors! Here’s what I kept seeing all over the log file:

I StCommunity 23/Nov/11, 04:40:57 Logged in to server 10.10.10.10

E StCommunity 23/Nov/11, 06:02:22 Connection broken to server 10.10.10.10, reason 0×80000224

W StCommunity 23/Nov/11, 06:02:22 Logged out from server 10.10.10.10, reason 0×80000224.

Now, I can’t read Sametime error codes and remember what they all mean by heart (not yet anyway). So, I did a quick Google search and found Technote #1098479 titled “Explanation of error codes associated with Sametime Community Services” (here’s a link).

In that Technote, you can see that the error code #0×80000224 simply means “the connection has been reset​”.

That is not useful at all.

The Weird Solution…

At that point, I had received a call back from our Lotus Support Rep and had sent him the CommunityConfig.txt and Sametime.ini from all 3 Sametime R8.5.2 classic servers in our company.

While the rep was busy looking for signs of trouble in the CommunityConfig file, I tried to contact our Notes Admin in Europe to ask him what he had tried so far. I couldn’t see him online but I tried to find him by name by using the search tool in the Sametime embedded client… here’s the weird thing I saw: he showed-up in the search results with his full canonical name instead of just his common name:

That was really weird. Normally, we only see 1 name (the 1st one)… we don’t see 2! The 2nd one even had the /UU (for Unseen University) listed twice.

Right away, I new something was wrong with the configuration of the Sametime Directory Service.

So, I opened the Sametime.ini of the server and found this little gem … there was 2 “[Directory]” section in the Sametime.ini of the server… and the 1st one was empty! Doh!

In the end, to resolve that weird “0×80000224″ error code, I simply had to remove the duplicate “[Directory]” section and reboot the server.

After that that server came back online, I restarted the ST Directory service on the 2 other Sametime servers and presto… all our users could see each other again.

Basically speaking, when a Sametime service loads, it reads the Sametime.ini from top-to-bottom. If it finds a section, it reads what’s in that section and loads according to the config of that section. It then stops and does not bother to check if there is another section below with more or different configuration settings. It’s completely logical and this is not a bug by my standards.

Now, why did the server work before without any problems with that duplicate section is a mystery to me… I’m just glad it’s back online and that all users can see each other now.

Conclusion?

Well, I don’t have much to say except that the error code is really cryptic and doesn’t help at all to explain what’s going in the “back” of the Sametime server.

I’m sure that if we had cranked up the debugging levels to crazy levels we would have eventually have found something… but how long would that have taken? Probably a few days… my track record with Lotus Support lately hasn’t been really good. I’m sure others have had much better experiences than me… I’m just unlucky.

Anyhow, I hope this helps someone, somewhere…

Thanks for reading and happy Thanksgiving to my colleagues in the USA!

Marc

To Paraphrase Professor Hubert J. Farnsworth…

Good News, Everybody! Lotus Support provided me a new Hot Fix for Lotus Sametime R8.5.2 Classic to, hopefully, help resolve my “Timed Out” issues that we’ve been seeing for the past two months.

The hot Fix number is NTOR-8N5C2Y in case anybody needs it. The Lotus Support Rep who sent me the HF wrote this in his email:

“While we are awaiting that data we have an update to the hotfix you are running on your servers to reduce the amount of traffic being sent by Sametime. NOTE: This fix MUST be applied on all three of the servers.

This is not a direct resolution to the issue you are facing. It may resolve your issue and it may not but it is a problem we have discovered and we are working to eliminate all problems to work past this issue.”

So… what do I have to loose? Answer: not much!

How To Apply The Hot Fix…

Well, Sametime Hot Fixes must be installed “manually” … so it goes like this:

1.Stop the Sametime server (duh!)

2. Back-up the following three StBuddylist jars that are located under Sametime binary folder (i.e. C:\Lotus\Domino usually):

BLBackwardCompatibilitySA.jar

BLFrontEndSubscriptionSA.jar

BLPresenceServicePublishSA.jar.

3. Extract the zip file that you received from Lotus Support and copy/paste the three jars contained in the HF into Sametime binary folder (i.e. once again in C:\Lotus\Domino normally).

4. Make sure that the three new Buddylist jar reside under Sametime binary folder (i.e. be paranoid and double-check that your copy worked!):

BLBackwardCompatibilitySA.jar

BLFrontEndSubscriptionSA.jar

BLPresenceServicePublishSA.jar

5. Start Sametime server (and start praying to the Deity of your choice).

After that… rinse & repeat the same steps on all your other Sametime servers.

How To Check If The Hot Fix Is Installed?

Well, after this jar-file-switch-a-roo, you might go full-paranoid and wonder if the Hot Fix is really installed. To verify that, you simply need to open up the latest Sametime_YYMMDD.log file on your server and check for 3 specific lines. If the lines below are there, then the Hot Fix is installed correctly:

I STPresenceCompatibility 23/Nov/11, 06:51:14 Started, version 8.5.2.04 – Hotfix # NTOR-8N5C2Y (SPR #GMOH8LMLKD, SPR #NTOR8KJJPA,SPR #PPLV8KJ9J3,SPR #NTOR8N5BQW

I STPresence 23/Nov/11, 06:51:15 Started, version 8.5.2.04 – Hotfix # NTOR-8N5C2Y (SPR #GMOH8LMLKD, SPR #NTOR8KJJPA,SPR #PPLV8KJ9J3,SPR #NTOR8N5BQW

I STPresenceSubscriptions 23/Nov/11, 06:51:16 Started, version 8.5.2.04 – Hotfix # NTOR-8N5C2Y (SPR #GMOH8LMLKD, SPR #NTOR8KJJPA,SPR #PPLV8KJ9J3,SPR #NTOR8N5BQW

It’s that simple to check if the Hot Fix is installed.

Conclusion?

Well, I don’t have much to say right now… I plan to install the Hot Fixes on my 2 other Sametime Classic R8.5.2 server a little bit later today (I can’t do it during business hours). After that, I’ll be playing the “wait-and-see” game and I’ll be on the look out to those “Timed Out” issues.

So in the meantime… thanks for reading!

I hope this helps someone, somewhere…

Marc