Function1 - AquaLogic Blog

The Function1 Holiday Enterprise Software Shopping Guide

Thu, 04 Dec 2008 16:03:10 -0500

Howdy all. With the holiday season upon us and everyone getting inundated with offers of bargain flat screen TVs at your favorite big box store and 10 spatulas for the price of 9 down at Spatula City, your friends at Function1 hope you'll save some of your hard-earned budget with this cost-saving tip: Make your software purchases at the end of a vendor's fiscal quarter.

Here's a not-so-well-kept secret about the software business: software and maintenance is a high-margin business for vendors. This means that there is no list price on that shiny new piece of software X that you're thinking of buying..."list price" is just a starting point for negotiating, alot like the tag in the window of a used car. What's more, most software sales folks have a sales target each and every quarter. The difference to a salesman between beating and missing their quarterly number is pretty significant. Beat their number and they:

Likely get a trip on the company to some exotic resort where they sip on delicious fruity drinks.
Buy that new convertible weekend car that they've been eying.

Miss their number and they can expect:

Lot's of yelling from angry VPs
Lot's of meetings and "planning sessions" with aforementioned VPs on how they're going to make their number next quarter
Perhaps start looking for a new job

The upshot here is that, come end of the quarter, if a vendor isn't meeting their financial targets, it's firesale time. In short, you have leverage to negotiate a much better price for your software. Say, for instance, you've received a quote to buy a site license for some software at $500K + 20% annual maintenance. Let's also say that, come end of the quarter, your favorite salesman is $300K short of meeting his sales target. Let's also say that the economy is currently in the toilet, and the vendor you're working with is in danger of missing the financial target they've given to Wall Street. All these factors add up to a perfect storm of you saving a bunch of cash. If you're not afraid to say, "No" to a price, there's a good chance you get the $500K software for $300K...if you buy before the end of the quarter. So, not only have you saved $200K from your budget immediately, but you're also realizing an annual maintenance savings of $40K...that's real money. Money you can use to:

Take your team on a trip to some exotic resort where you sip on delicious fruity drinks.
Throw a kick-ass holiday party
Hire your favorite consultants at Function1 to give you a hand in getting the most out of your new software purchase

That's about all for now. Hope you find this tip helpful, and that it saves your some $$$. If you're really in the holiday spirit, feel free to send us a check for 10% of any money you save :)

Welcome Sandeep Khaneja and Hani Atalla!

Wed, 26 Nov 2008 23:18:42 -0500

Function1 is a growing company, but not too big that we can't take a minute out of our regular blog routine to welcome our two newest team members, Sandeep Khaneja and Hani Atalla.

Hani and Sandeep have actually been with us for several months, so my apologies for the late public welcome, but we're thrilled to welcome them to the team and have them work with our clients delivering elite AquaLogic consulting and product expertise!

Sandeep is a Plumtree/BEA Alumni with years of experience helping clients like yourselves realize their vision of a best-of-class end-user focused portal, with a special focus on strategic business planning, health checks, and technical optimization of the rest of the AquaLogic product stack.

Hani is a multi-year ALUI consulting veteran with a solid Java and .NET application development background for ALUI clients, and has been instumental in developing Watcher, our newest product - an ALUI monitoring an alerting application.

Welcome guys!

The Exciting Conclusion of Stack Wars

Sat, 22 Nov 2008 18:14:34 -0500

OK, now that you had your dinner and ate your meat, it's time for the pudding. This conclusion to Stack Wars just ties up a few loose ends and walks through a "real-world" example of analyzing stack traces. Have fun...

Making Thread dumps easier to view

Seeing as how most people's idea of a good time isn't sifting through pages upon pages of JVM stack traces, several good people have developed tools that make it easier to view and analyze complex thread dumps. Personally, I like and use the IBM Alphaworks Thread and Monitor Dump Analyzer. Pretty straight forward to use. Just save your thread dump off to a file and open it up in the Analyzer for a pretty view of all your threads that looks like this:

I've found this tool useful when I'm trying to look for patterns across many threads in a thread dump. Give it a try, you might like it.

A real world example of how all the rambling I've just gone through came in handy

Long ago, in a galaxy far, far away, I was working on a Plumtree consulting project with a large company. These folks had been having problems with their portal deployment. Periodically, the portal would start eating up 100% of the server CPU, and would never release it until the portal application server was bounced. During the time that the server CPU was pegged, users couldn't access the portal...the customer was not happy. It was one of those big deals where VP-level people at the customer were yelling at VP/Executive level people at Plumtree on a daily basis. So here I am in this high stress situation. I arrive onsite with the customer and, for three days, nothing happens. The application doesn't freak out, and all is fine. At this point, I'm starting to get cautiously optimistic that maybe we won't see any problems during my remaining two days onsite, and somebody else will have to deal with the problem later :) But, alas, it wasn't to be. About halfway through my second to last day with the customer everything went to hell. All of a sudden nobody could access the portal, so we start debugging. Sure enough, first thing we see is that the portal process is eating up 100% of the server CPU. These folks were running on a Unix platform, so I ask them to try:

kill -QUIT <portal pid>

After explaining for 20 minutes that, no, this command won't kill the process, the server ops folks generated a thread dump and sent it over to review. After about 5 minutes of looking at the portal thread dump, it was pretty obvious that something was amiss with the portal code. The stack trace for EVERY SINGLE THREAD (All 100 or so of them) was exactly the same. They were all stuck running the same native method. Just to be sure this wasn't some freak anomaly, we generated another thread dump 15 minutes later, and, yep, all the stack traces looked the same. So this tells us that there's a problem with some native C code being used by the application...a good start. Unfortunately, since the problem was in native code, we couldn't get a full stack trace from the JVM...it only goes so far as to let you know that it's making a native call. So we dig one level deeper and trace the portal process. Running a trace/truss on a Unix process spits out all the system level calls that are being made. Wouldn't you know it, when we looked at the trace output, 99% of the calls being made were:

poll(0)

Now, I wasn't an expert at interpreting this level of data, but I knew enough that "poll" had something to do with sockets, and that there definitely shouldn't be so many of those calls. Long story a little bit shorter, after a bunch of conference calls with Plumtree support and engineering, where we shared the data we'd gathered from the thread dumps and process traces, the engineering team found a very low level bug in the software and shipped a fix out to the customer...tragedy narrowly averted.

Conclusion

Did you really make it this far? If so, thanks for sticking with me, and I hope you found something of use in this long, winding post. If you're ever stuck trying to make sense of a thread dump, feel free to drop us a line or post a comment here...we'll do our best to help.

Take care, and remember, when dealing with production troubleshooting, "Do...or do not, there is no try". (This doesn't have really have much to do with the post, but I'm kind of tired and pulling at straws for good ways to tie the Star Wars metaphor back in). Anyhow, as always, thanks for reading...see you next time.

Stack Wars III (Or VII, depending on how you count): Return of the Stack

Tue, 18 Nov 2008 19:11:01 -0500

READER BEWARE: This got long and geeky, so make sure you're really trying to avoid doing "real work" before you sit down to read.

Welcome back all, and thanks for joining me for the final installment of a three part series on decompiling Java code and analyzing stack traces. If you're interested in the back story, you can read about decompiling Java here, and analyzing basic stack traces here.

When we last left off, our hero was frozen in a block of Carbonite and left to his doom. Err...sorry, wrong story, let me start again.

When we last left off, we walked through analyzing a simple stack trace to run down a bug in a standalone Java program. This is all fine and good, but what happens when you're running a Java application server (like Tomcat or Weblogic) that has hundreds of concurrent threads running tens of different web apps? And you're just trying to figure out why your particular web application is hung?

Well, if you're lucky, the developer of your web application was a good boy/girl, and they're logging stack traces for you in a log file somewhere. If this is the case, then you can open up the log and analyze the trace like we did the last go round.

Often times though, you're not so lucky, and you have to dig a little deeper to figure out what's going on. Say, for instance, your web application just starts responding very slowly. You see from access logs that responses are being served back to users, but they're about 5 times slower than normal...WTF? Or, what if your application server starts pegging the box at 100% CPU...what to do? Or, out of nowhere, your app server starts throwing Java.lang.OutOfMemory exceptions...Gah!!! Sadly, these nebulous problems seem to happen in a production environment more often than most of us would like to admit. And when they do occur, it's usually a high stress situation because there's probably a production outage and nobody really knows why. So, how do we find a better fix to the problem than the traditional, "Let's just bounce it and see what happens" response? Why, we use "The Force", of course. Except in this case, "The Force", is just a set of debugging tips that I'm getting ready to share with you as follows:

Take a deep breath and don't freak out when a bunch of people start yelling.
Understand the severity of the situation. Figure out how much downtime you can tolerate for debugging before things really hit the fan.
Set expectations. Let people know when they can expect the issue to be fixed. If you know a bounce will temporarily fix the problem, then set a drop-dead time for debugging and schedule a bounce. Let folks know that if you don't find a root-cause of the problem, that they should expect to see the issue again.
Gather as much information about the problem as you can. What, specifically, are users experiencing? Can you reproduce? What are the symptoms? What log messages do you have? What does the server environment look like? etc.
Try to relate this issue to something you've seen in the past. Does this look like a problem you saw last week or last month? What did you do to fix it then? Why is it popping up again now?
Eliminate possible external causes. Is this actually a network problem in disguise? Is the database acting up? Is some other process on the server eating up CPU? Is the server constantly swapping because it doesn't have enough RAM to handle all its business?
Eliminate environment changes as possible causes. Has the server environment changed recently? Could this be causing your problem?
Look in earnest at your application server process. Are we bumping up against the Java max. heap size? Are we seeing lots of garbage collection? Is our thread pool exhausted? Is our JDBC pool maxed out?
Take a look inside of the application server JVM. Generate a thread dump to get a snapshot of how your application server is behaving. Analyze the stack traces in the thread dump, and look for points of interest, like...race conditions, stuck threads, deadlock, etc.
Open a ticket with the software vendor (If applicable)
Pray the issue fixes itself and doesn't come back
Find a new job so you don't have to deal with these problems anymore

Read on for more fascinating details about The Stack Trace...

AJAX Refresher

Fri, 14 Nov 2008 23:23:00 -0500

It's been a while since we touched on AJAX, but a question came up recently about it an I thought it might be good to review. AJAX, or "Asynchronous JavaScript and XML", is a way for portlet developers to create rich Web Applications that don't require the entire browser page to refresh to update content. This is done by making asynchronous calls to the server and updating content within the page itself. With the AquaLogic portal, this means that portlets can dynamically update content in <div> tags by requesting new content without having to refresh the entire page (and other portlets on the page). It's a pretty simple concept; in many cases you can accomplish this without having to even change any code - you can just specify "inline refresh" on the Web Service and the portal will automatically rewrite the HTML links on the page to make AJAX calls:

The HTML rewrites cause the browser to make the HTTP request "behind the scenes", and when a response comes back, the portal refreshes the content inside the portlet <div> tag.

But there are some things to know about this AJAX stuff, so here are a couple of refresher points about AJAX:

1) The response to an AJAX request is basically just a text string to a browser, and it's up to your JavaScript to interpret it. Often you do something like:

document.getElementById("responseDivTag").innerHTML = response.getResponse();

... to refresh content. But note that this doesn't tell the browser to "process" the response - specifically, JavaScript that comes back in the response won't run, because all we're doing is setting the HTML to a string that comes back from the server. In order to run JavaScript in the response, you should look into the JavaScript "eval()" method, which will take a string returned from the server and run it as JavaScript. Just make sure you don't include the <script> tags in your response if you really are returning JavaScript and are parsing it as such.

2) The response does not have to actually be HTML! It's just a string to the browser, and you can do anything with it. The most common use (which all of our products use) is to return JSON, or "JavaScript Object Notation", which can then be treated as objects that your script can handle however you want. Let's say you just want to know if there was a success or failure: you could literally just return a "0" or "1" in your response and write something like:

if (response.getResponseText.equals("1"))
   alert("success!");
else
   alert("fail");

Obviously, this just barely scratches the surface on AJAX, and you can rest assured that you haven't heard the last of it. AJAX is the cornerstone of pretty much all future Oracle portal technologies, and if you're a web developer who's not all that familiar with it, trust me: you will be soon.

The Stack Trace Strikes Back

Mon, 10 Nov 2008 23:56:00 -0500

Howdy all. Welcome to part two of three of what was originally conceived as a one part series. It's entirely possible that I'll get all George Lucas on you years from now and produce some more of these posts that are a complete letdown and affront to your childhood memories, but I digress. For now, rest assured that this post will knock your socks off as a follow-up to my last tidbit on decompiling Java code.

Without further ado, I give you...Stack Wars II: The Stacktrace Strikes Back (I'm completely aware that I'm abusing the metaphor here, but isn't that really what blogging is all about?).

Standard disclaimer: This post is intended for a technical audience with a focus on production support. Also, everything here is Java focused, but you can certainly apply some of the concepts in a .NET environment as well...you'll just have to create your own screencaps to replace the examples I've included below.

So, what is a stack trace, and why should you care? Well, one question at a time please.

What is a stack trace?

Wikipedia says a Stack Trace is, "A report of the active stack frames instantiated by the execution of a program." Now, I vaguely understand the Wikipedia definition, but I have also have a computer science degree from a second tier state university, so let me try to translate for those of you who were smart enough to get degrees in something besides CompSci: a Stacktrace is a snapshot of a program's behavior at a point in time. In the Java world, a stack trace will tell you which method was being executed at the time the trace was generated, along with its complete call stack, and usually line numbers as well. Take a look at the following simple stack trace below as an example:

Why should you care?

Good question. I'd venture a guess and say that about 99.99% of the world doesn't need to know nor care about stack traces. But here you are reading this post none-the-less, so here's why they're important:

1) Good programmers almost always print stack traces out in log files when an error in a program occurs. This gives us a useful tool to track down bugs. Whether you're just reporting information to a support team somewhere, or getting a little sassy and trying to fix a problem yourself, the stack trace is like a map for finding treasure buried deep in code. Except that instead of finding actual treasure, you're just finding a logic error. And instead of getting rich, you just get to complain about a problem, and maybe fix it.

2) You can tell the JVM to generate a stack trace for a running process. Doing so allows us to take a snapshot of the JVM at an arbitrary point in time, and see what all its threads are up to. This is useful when trying to figure out why a process (Tomcat for instance) is zombied (i.e. it's running, but not responding to requests), or when you're trying to fix deadlock issues, which are particularly difficult to run down.

Hit the jump for learning more about reading and interpreting stack traces!

Cool Tools Part XVI: My Love Affair With JAD

Mon, 13 Oct 2008 16:53:17 -0500

Hi, my name is Brian. I like sunsets, long walks on the beach, puppies, and de-compiling Java code. If you have similar interest, please read on. On the other hand, if you're not a Java programmer, this post might not be for you.

So of all the cool tools out there, JAD is, without a doubt, my absolute favorite...and it's not even close. One of the beautiful things about Java is that the byte-code interpreted by the JVM can be reverse-engineered into source code. JAD, or JavA Decompiler, is a tool for doing just as the name implies: decompiling byte code into source. But so what? Why would this possibly be useful? Well, here's a list of just a few things that I've done by de-compiling code:

Found software bugs
Saved about 5 weeks of hassle with BEA support by pointing them to the line of source code that is causing a bug
Re-compiled bug fixes back into code as a band-aid solution to a problem (Note that this probably voids any/all support agreements that you've ever had with any one)
Understood how/why a piece of software works like it does (I.e. "Why is Analytics deleting all a bunch of historic information when I run a specific job...there's no document on it")
Found cool/un-documented software features (Again...no support, but playing around with stuff is fun)
This one is my favorite: Long ago, for "Bring your kids to work day", I was on the hook for talking to a bunch of 12 year olds about programming. So I de-compiled a Ms. Pacman applet, found the code that does collision detection, modified it to not work, and re-compiled the new code back in. The net result was a Ms. Pacman who could run through ghosts, and a bunch of kids learning how "If" statements work.
Also, this guy I know might have one time de-compiled a VI plugin for Eclipse to take out the license key check. Don't worry though, I quickly told him how he was a bad boy, and he's been on the straight and narrow ever since.

OK, sorry for the rambling there, but just wanted to give you a flavor of the stuff you can do by de-compiling code. More generally though, if you're a programmer at heart, it's always a good idea to look at source code, just to understand how things work. Of course there are fringe benefits too:

Fellow portal geeks marvel over how "smart" you are because you "de-compiled" the portal and found some problem.
You'll get to complain about what idiots the people who wrote the code are
Men will want to be you
Women will want to be with you

So, let's look at a quick example, shall we? JAD is a text-based utility, so get ready to fire-up your DOS prompts and/or Shell windows. I'm writing this post on a PC, so my examples will be through DOS, but UNIX folks, just make the normal DOS->*NIX changes, and you'll be fine.

Let's assume that I've been trying to get a piece of proprietary software to run for a while now, and every time I start it up, I get a stacktrace like the following:

Now, you could argue that I should know better than to purchase a piece of software named, "Broken", but I'd argue right back that you're getting in the way of my example.

Want to see JAD in action? Join me after the link for all the details!

Solaris, PTSocketSelector, and sun.nio.ch.PollSelectorProvider...Oh My!

Tue, 07 Oct 2008 12:01:53 -0500

Howdy all. After a long hiatus I'm back to blog your socks off with some technical minutiae that will save a few of you lots of headaches, and help the rest of you get a good night's sleep.

Long Story Short (i.e. "Just the useful info, please")

There's a bug in the JDK that causes problems when setting up certain types of Java server socket constructs on some Solaris 10 boxes. This bug will likely manifest itself in one of the following ways in your ALUI environment:

1) You're running the ptlogging utility and see your logs filled up with something like:

PTSelectorThread_17679958

com.plumtree.openkernel.impl.openhttp.core.network.PTSocketSelector     Unexpected exception.

java.io.IOException: Invalid argument

        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)

        at sun.nio.ch.DevPollArrayWrapper.poll(Unknown Source)

        at sun.nio.ch.DevPollSelectorImpl.doSelect(Unknown Source)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

        at sun.nio.ch.SelectorImpl.select(Unknown Source)

        at sun.nio.ch.SelectorImpl.select(Unknown Source)

        at com.plumtree.openkernel.impl.openhttp.core.network.PTSocketSelector.run(PTSocketSelector.java:400)

        at java.lang.Thread.run(Unknown Source)

2) Portal starts up fine, but it can't connect to any remote servers, giving you the same stacktrace as above:

java.io.IOException: Invalid argument

        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)

        at sun.nio.ch.DevPollArrayWrapper.poll(Unknown Source)

        at sun.nio.ch.DevPollSelectorImpl.doSelect(Unknown Source)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

        at sun.nio.ch.SelectorImpl.select(Unknown Source)

        at sun.nio.ch.SelectorImpl.select(Unknown Source)

        at com.plumtree.openkernel.impl.openhttp.core.network.PTSocketSelector.run(PTSocketSelector.java:400)

        at java.lang.Thread.run(Unknown Source)

So what's a good god-fearing person like yourself to do? Well, if you act now, we'll send you TWO workarounds for the price of one:

1) Have an SA up the hard File Descriptor (FD) limit on the server to 8193 or greater by editing /etc/system to have the line:

set rlim_fd_max=8193

Note: You'll need to bounce the box after this change. You can then verify it worked by running:

ulimit -n -H

Which should return a number >= 8193. Sadly, this approach will probably have the SA asking you why you want to make the change, which means you'll have to read the technical details below....so onto option #2

2) Tell the appropriate JVMs to use a different Socket Selector configuration. You do this by passing the following option to the JVM:

-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider

Depending on what ALUI component you're updating, you may pass this option in different ways. For instance, if you're dealing with one of the back-end servers (Collab, Studio, etc), you'll want to update wrapper.conf to add additional arguments like:

wrapper.java.additional.7=-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider

Note: Replace "7" in the above line with the appropriate number for your wrapper file.

Or you may just need to update a shell script somewhere that's kicking off Tomcat/Weblogic/etc. Note that these scripts all have their own Shell variables for holding additional Java arguments, so just look through them and update as appropriate. If you have problems, feel free to post questions here and we'll do our best to help out.

Long Story Long (i.e. "I'm kind of a geek, and I'm sitting at work with nothing else to do, so give me the details")

So ALUI uses the NIO java packages that were introduced in Java 1.4. FWIW, I always thought NIO stood for Non-blocking IO, but a little Googling reminds me that it's actually New IO...silly me. In any case, the NIO packages let you do some cool things with sockets to more efficiently manage high-volume connections. The under-lying problem you're running into is that out-of-the-box Selector implementation in the JDK uses /dev/poll to allocate 8192 File Descriptors (FD) for use by the selector, and 8192 exceeds the nofiles (Number of File Descriptors) limit on your server. So, you can either bump up the server FD limit ala work-around #1 above, or tell Java to use a different selector implementation that doesn't allocate all those FDs ala option #2 above. If you're interested in more detail, you can find the Sun bug on the issue here.

Until next time...thanks for reading!

AquaLogic: Alive and Kicking

Fri, 03 Oct 2008 19:14:52 -0500

We just got back from OracleWorld last week, and wow! That conference is unbelievably huge, but very well-organized. We met a lot of current and past friends and colleagues out there, but a big part of our attendance was to get more insight into the future of the AquaLogic portal stack.

Once again, Oracle reaffirmed 9 years worth of support for the AquaLogic line, but their commitment to keeping Aqualogic as the primary code base wasn't as strongly expressed. As you may know, Oracle has the WebCenter Suite - of which ALUI (WebCenter User Interaction), will play a part. But Oracle's portal and BEA's portal will also remain in this space. The current plan is to take most of the existing integration products and portals and fit them together in a more standards-based suite of producers (Oracle Services) and consumers (portal products), so you'll be able to leverage all of the cool new services that Oracle is working on with any of these portals.

They're also working on this thing called Oracle WebCenter Spaces, which is a fantastic-looking UI for creating "portals" with JDeveloper and Oracle's Application Developer Framework. This thing was slick - we went to a lab where we created a rich mock-customer UI which integrated elements of Oracle's Content Management system (which'll replace Publisher) using JDeveloper and just dragging-and-dropping widgets onto a page and wiring the objects together with property pages. The thing felt more like BEA's portal (which was more developer-focused) than AquaLogic's (more configuration than development), but it was still pretty impressive.

We were told all of those rich components will be surfaced through the AquaLogic portal as well via WSRP, so in theory, ALUI customers aren't left out in the cold as these rich components are developed. As Oracle fine-tunes the ultimate portal strategy, they're also developing these components as "WebCenter Services", which are basically standards-compliant portlets that will be available to all portals. So we should be getting the best of all worlds - existing portals plus all the "generic" web services (like discussion groups/collab, and the new Enterprise 2.0 functionality).

The big outstanding question, though, is whether WebCenter Spaces will become the de-facto "portal" that gets the most development effort from the Oracle Team on a go-forward basis. Eventually these products will have to start consolidating, and time will tell whether that's nine years from now where each product has gradually started looking like the next anyway, or 2 years from now where more dramatic migration strategies will need to be put in place. My take? Either way, it's no big deal. We Plumtree folks have had some pretty major upgrade/migrations to go through before, and we were always taken care of. Remember how hard the 4.5WS -> 5.0 migration was? Well, we haven't had a major game-changing upgrade since then (the 5 -> 6 upgrade was pretty clean, but was more evolutionary than revolutionary). If Oracle does go with a big migration strategy from the ALUI code base to something else, I'd bet that the migration won't be as complicated as the 5.0 migration was, and the rich new set of functionality available will make it worthwhile.

The important thing here is: don't panic! ALUI will be around and recognizable for a LONG time, and judging by what we saw at OracleWorld, there are lots of exciting new technologies just around the corner that will make it even more valuable.

Remotely Reboot Windows Servers

Wed, 01 Oct 2008 22:44:00 -0500

You may never need this one, but when and if you do, you'll be glad you've got this tip in your back pocket. No doubt, you never actually sit in front of a console for your AquaLogic Servers; if you're on Windows, you're using Windows Remote Desktop.

Occasionally, you need to reboot those servers, and you do it through the remote desktop. But once in a blue moon (or more - it's happened to me on a half dozen machines at one client and 2 machines at another in the last month), you go to reboot and Remote Desktop never comes back. Typically, this is because Windows has started shutting down and killing windows processes - including Terminal Services. But for whatever reason, Windows doesn't reboot, and Remote Desktop is no longer available because the process is gone and isn't restarted.

The solution: remotely force the box to reboot again. Simply run:

shutdown /r /f /m \\servername

... from any of the machines in the subnet. You're telling the OS to force (/f) a reboot (/r) on a remote machine (/m). Works every time the OS is available but the Terminal Service is dead - check out the MS article for more info on this command.

Beware the Security Propagation Bug(s)

Sun, 28 Sep 2008 22:54:10 -0500

We've warned you before about ACL propagation when you're changing the security in ALI. Heck, we even created a product to ease the pain of this important task. Today's bug is about another issue with security propagation.

Well, it's actually 2 bugs (maybe 3). Let me explain:

When you answer "yes" to that question about security propagation, a job is created. Here's the problem: the job is run as the user who created the folder, not the user changing the security. What if you later delete that user? Well, bug #1: automation server is hosed. You're going to get an error like this:

The Exception says "*** Job Operation #1 of 1 with ClassID 20 and ObjectID 898 cannot be run, probably because the operation has been deleted.", and the error's wrong (because it says "probably", I won't count this as a bug). The real problem in this case is that the folder's OWNER has been deleted, not the operation itself.

This gets me to Bug #2: when you delete a user, they're removed from all groups, but apparently they're not removed as OWNERS of any of the admin folder objects (and how could they? What should they revert back to?). Obviously, this is what causes the problem with automation server. If this one was fixed somehow, bug #1 would become irrelevant.

Bug #3 (this one I haven't confirmed yet, but is likely to exist given the testing and logs mentioned above): suppose Joe Content Manager creates an entire structure of folders. Then Joe's boss decides he shouldn't be an admin for one of the child folders (say, an "Executive Committee" folder) and removes his access. He changes the security and propagates it on that "Executive Committee" folder. Later, he wants to add Mary Executive to have privileges to the folder. He changes the ACL again and chooses to propagate. Here's the rub: because the job is run as Joe, and Joe no longer has access, the job fails, and Mary can not be added. In fact, it's likely that Joe can't even be added back, so that ACL is completely frozen unless the admin goes through every object and re-adds Joe back (or changes the owner of the folders through the DB). Have I mentioned LockDown?

Anyway, want the SQL Scripts to fix the owners on the various folders? Hit the jump.

Crazy Database Growth? Check your PTJOBLOGS table.

Thu, 25 Sep 2008 10:54:00 -0500

This is a little old-school, but still a very relevant tip. This problem has been around for all of the ALI 6.x days (including 6.5), and if I remember correctly, even the Plumtree 5.x days. Basically, the automation server logs its activity to the PTJOBLOGS table, and if you've got jobs that are logging in verbose or very frequently, this table can become HUGE: I had a recent customer whose database had grown to over 80 GB because of this table.

Basically, to prevent this table from growing astronomically big, make sure:

You don't have jobs that are running way too often (once a minute typically means a job is running - and logging - constantly)
You don't have verbose logging for any jobs running in the portal, and
That your portal configuration is set to only save a reasonable amount of job log data. By default, the portal will keep around 60 days worth of logs, which is a pretty big number. If you ask me, any job logs older than 7 days are worthless because jobs always run more than once a week. But, of course, you're not asking me, so I'll just tell you how to change this setting and you can decide for yourself. The configuration setting isn't available through the UI; instead you have to tweak the database. Specifically, you have to update the PTSERVERCONFIG table in the ALI database with SETTINGID=15. Set the value to whatever you think is appropriate based on your use of the job log:

Finally, what happens if you've already got a PTJOBLOGS table that contains 200 million rows? Here's a tip that I got from our friend and client, Mike Jones at MedSolutions, Inc.: if you run an amateur update like "delete * from ptjoblogs", the update will take forever because all transactions will be logged. If you run "truncate table ptjoblogs", on the other hand, you're just dropping the table and don't have to wait ridiculous amounts of time for the update to happen (not to mention the amount of database log space you'd be consuming).

ALI Collaboration 4.5 can email you if email is broken

Mon, 22 Sep 2008 13:06:00 -0500

Here's a little feature that some of you may find useful: Collaboration Server 4.5 can send you an email if it can't connect to the new Notification Service. For those of you that have countless problems with the old Notification Server that shipped with Collab 4.2 and earlier, this is a must-have feature.

Just go to Administration: Select Utility...: Collaboration Administration: Collaboration Notification, and enable Health Monitoring:

It works, too (assuming your SMTP server doesn't require authentication for internal addresses). Collab is even kind enough to mark the mail as "High importance":

On the other hand, for those of you that don't appreciate this new little nugget of functionality, consider this irony: Collaboration 4.5 uses email to tell you when email is broken. Cosmic, man.

Set your Login Token for AquaLogic Notification Service

Fri, 19 Sep 2008 21:26:00 -0500

STILL working on Collab 4.5 RSS feeds... Today's tip will either be completely useless to you, or save you a ton of time. If you ever see the following error:

collab.server.administrator CNS.SECURITY com.bea.notification.security.SecurityManager
Unable to authenticate user with token '1|1288135304|xarwmCj1FbIzv/Mo/yyd0tEjkgI='

Don't bother looking at the Security And Directory Service (really, what's that thing for anyway?). The following screen shot explains it all:

Open your portal database, query PTSERVERCONFIG ("select VALUE from PTSERVERCONFIG where SETTINGID=65"), and stick that value in the "Message authentication code seed value" field on the "Aqualogic Notification Service"/"Login Tokens" page.

Maybe when I'm much less annoyed at this whole process I'll dig more into the SAML2 Token Type. And that elusive "Security and Directory Service", which still seems to provide no more valuable service than being a red herring for bizarre portal issues.

ALI 6.5 Configuration Manager

Tue, 16 Sep 2008 07:50:00 -0500

When I first introduced the Configuration Manager, I wondered aloud how it would work in a multi-system environment.

Well, it turned out the the Configuration Manager isn't that complicated after all: it's pretty much an interface for editing a configuration file on disk, so it does only apply to the system you're running it on.

The file can be found at bea\alui\settings\configuration.xml, and can be edited directly as easily as you would through the Configuration Manager interface (with the exception of passwords, which are encrypted):