Function1 - AquaLogic Blog

Search Index Exceeds 10GB Logical Limit: KABOOM

2010-08-20T00:28:32Z

A small nugget of information is hidden in a technical white paper for WebCenter Interaction (Page 7). Oracle suggests that “Each (search) node can contain up to 10GB of index data for 32-bit operating systems and 100GB for 64-bit”. What the white paper fails to mention is what happens if this limit is exceeded?

In pseudo-Function1-mythbusters fashion, we tested this rumor inadvertently for you. Our symptoms started when file and web crawlers were intermittently failing. Search and snapshot queries were still working as normal and no errors were showing in the UI. The job log mentioned two errors:

-2147204349 – Error in function PTIndexer.CommitAll (): -2147204349 – Error reported by server: error writing index request to disk: FILE_WRITE_FAILED (-2147204349)

…and a similar error mentioning a FILE_READ_FAILURE on index requests.

Further investigation uncovered a number of similar error messages in the search node logs under \ptsearchserver\[version]\[node]\logs

<entry timestamp=”2010-07-28T08:06:58+04:00″ thread=”624″ >
<error>Failed to write to file: \\..\requests\1_0_256\2761711648.15244.488488.delta</error>
</entry>
<entry timestamp=”2010-07-28T08:06:58+04:00″ thread=”624″ >
<error>Error FILE_WRITE_FAILED in ..\..\src\IndexRecord.cc at line 192 (system error code 6: The handle is invalid.)</error>
</entry>
<entry timestamp=”2010-07-28T08:06:58+04:00″ thread=”624″ >
<error>Error FILE_WRITE_FAILED in ..\..\src\IndexQueueMonitor.cc at line 104 (system error code 6: The handle is invalid.)</error>
</entry>
<entry timestamp=”2010-07-28T08:06:59+04:00″ thread=”624″ >
<error>error writing index request to disk: FILE_WRITE_FAILED</error>
</entry>
<entry timestamp=”2010-07-28T08:06:59+04:00″ thread=”624″ >
<error>Error FILE_WRITE_FAILED in ..\..\src\QueryParser.cc at line 3049 (system error code 6: The handle is invalid.)</error>
</entry>

Before realizing the search index had grown to 10GB, we attempted to restore from a prior checkpoint. On a side note, scheduled checkpoints are a lifesaver in search; if you don’t have regularly scheduled checkpoints open another browser tab to your Portal and schedule a checkpoint. Restoring the checkpoint worked for a short time until Search Update started adding information to the index. Once again, errors started popping up in the search logs.

After reviewing the search node indexes, we determined that a new partition would reduce the index size of each node from 10GB (two partitions) to 6.6GB (three partitions). Repartitioning creates the new partition alongside the old one in the node, and our index grew to around 14.5GB. If Search were a cement truck, it would look something like this: Mythbusters Cement Truck Explosion

Search no longer accepted requests, snapshot queries were empty, and no job could read or write to the search index.

Resolution:
The only way out at this point was to purge all the nodes using cadmin -purge, delete all checkpoint history, partition information, and request information. Here are some detailed steps on how this was done:

Use this method if your Search Server will not start.

1.) Stop all search servers (if any are still running).
2.) Delete the contents of the following folders (not the folders themselves):

\alui\ptsearchserver\cluster\requests\
\alui\ptsearchserver\cluster\repartitions\
\alui\ptsearchserver\cluster\checkpoints\

3.) For each search server in the cluster, perform the following:

a. Go into the index folder of the node. For example, \alui\ptsearchserver\6.1\<nodename>\index.
b. The index folder will contain a folder named “1″. Open this folder and delete all its contents.
c. The index folder may also contain a folder named “2″. If this is the case, delete the 2 folder.
d. The index folder should contain a file named “ready.” Open it in a text editor and make sure it contains just the number 1. This “ready” file tells the node to look within the 1 folder for its content.
e. Open a command prompt within the 1 folder.
f. Run the following commands. The first command should create about 8 files. The second command should create about 7 files:

i. ..\..\..\bin\native\emptyarchive lexicon archive
ii. ..\..\..\bin\native\emptyarchive spell spell

We then repartitioned using the cluster.nodes file under cluster_home and the cadmin -repartition command. This created three empty partitions where a Search Repair could be performed.

Myth Confirmed:
Each search node can contain up to 10GB of index data for 32-bit operating systems. In fact, each search node should not exceed around 7GB-7.5GB so you have enough space to repartition the index without failure.

For those of you who like explosions, see the full Mythbusters Cement Truck explosion here.

PSA: Avoiding Catastrophic Failure

2010-06-25T15:20:23Z

The back story

I recently had the misfortune of being the victim of theft. As a frequent traveller, I keep many of my most important items (technology, travel documents, iPhone charger, breath mints) close at hand in a single bag. At a moments notice I can be out the door and on a plane to help my favorite portal customer. This particular theft reminded me that no matter how much I back up important documents, it doesn’t stop someone from stealing the physical backup media.

Years of experience (i.e. viruses, windows crashes, drive failures, drink spilling, and fumbles) have taught me to always backup. I carry an external harddrive along with my laptop that has a backup of my most important personal and professional documents. For the critical items, I also have a solid state media (flash drive) just to be sure. To make travel easier, I usually have said backup devices parked right alongside my laptop for easy access.

GASP!!! You could hear a pin-drop when I realized someone had decided my bag was much better looking in his/her possession. My immediate disgust wasn’t because of the $2,000+ worth of electronics I would have to replace, but the information on those devices. Countless hours of design, development, and documentation work were suddenly gone.

A Public Service Announcement from your friendly Function1 Consultants

As good netizens, we decided to offer some alternative solutions to this problem in the hopes that our beloved clients don’t have to feel the sting of regret.

Prey: They can run but they can’t hide

I was amazed by how little the police could do to retrieve my laptop and web stats like “97% of stolen laptops are never recovered” doesn’t make me feel any better. Prey is an open source initiative that installs a small application on your laptop. You activate the application via a web interface if your laptop happens to go missing. Once activated, Prey will send you screenshots, IP information, and (if you have a webcam) a picture of the user. If the laptop has wifi in use, Prey will also try to geo-locate your laptop by triangulating wifi access points. Best of all, its FREE!

http://preyproject.com/

Backup: You can’t steal the Internet

As high-speed internet and cloud computing have grown, online backup has become a realistic solution for joe consumer (i.e. me). I found a ton of information on solutions to online backup, but as a tech geek I wanted the coolest of the cool (and cheap too).

It took me a while to find an online backup solution that could meet all my needs as a tech-geek, gadgeteer, and professional IT type. I wanted sync tools, “set-and-forget”, lots of gigabytes, and maybe a cool iPhone app. Sugar sync covers virtually all of the features I could cook up in a backup solution, then throws in some neat utilities. The interface is extremely straight forward and I was up and running within a couple of minutes. Compared to other backup solutions, sugarsync wasn’t the lowest price per gb; but it has by far the most features.

Pro’s:

Choose any folder on your PC to sync with the “cloud”
Sync the same files with up to 5 different computers.
Real-time sync when you’re online
Version history
File resume, so you don’t have to wait for a sync to finish before powering down.
Online edit mode lets you edit from your personal sugarsync website
iPhone app gives you access to your files anywhere
256bit encryption

Con’s

Not the cheapest per gigabyte of storage. Microsoft Skydrive at $0 for 25gb was the largest free storage solution. Carbonite was a close second at around $55 for unlimited space.
Cannot sync with a local external harddrive. I was hoping I could sync my files with the “cloud” and also update a local copy for backup.

http://www.sugarsync.com

(I also checked out Microsoft Skydrive, Dropbox, Mozy, Box.Net, and Carbonite before deciding where to drop my $$$. Each had its short-comings and benefits.)

There are a lot of things I WISH I had done, so I hope my experience prompts our clients to keep all that hard work and sensitive information within reach no matter the circumstances.

Meta-Post #1: A Blog Post About Blog Posts

2010-05-01T14:55:27Z

I’m randomly up early on a Saturday morning and decided to use the time to put out this week’s belated blog post. There are a bunch of things I probably should be writing about (including the WCI->Sharepoint migration tool we’re building…look for more info soon), but it’s before 8 AM on a Saturday, and I’m just not up for writing about technology. So instead, you get this: a blog post about our blog posts.

One of the reasons I left BEA was because I was tired of the big-company atmosphere. Tired of the bureaucracy, of just being a number, of the corporate politics, and all the cons that come along with the positives of working for a large organization. All in all, I felt like being at BEA was just kind of blah. There wasn’t any corporate culture, and I didn’t identify with the company. Maybe these things aren’t that important, but I think they are. And it’s something we’re cognizant of at Function1: corporate culture and identity matter. It’s an interesting challenge to build a consulting company in which employees feel part of a bigger whole, and not just off by themselves in Anytown, USA billing hours. Likewise, deciding how you present your company to the outside world is an intriguing question to ponder. How do you differentiate yourself from the approximately 1.27 gazillion other nameless consulting companies out there? How do you want customers, partners, potential hires, and the general public to perceive your company? Hard questions to answer, but my gut tells me that the best answer is to present your company honestly, as you are.

Which brings us to the crux of today’s post. There’s been a long-standing on again/off again discussion amongst our circle of professional friends about the style and tone of the Function1 blog. Some people like it, and some people think it’s not professional enough. My take is this: other than our people, this blog is the most consistent public-facing component of Function1. And it’s an honest reflection of who we are. We’re a small company that’s damn good at what we do, and we have fun doing it. We work hard for our customers and ourselves, but we don’t take ourselves so seriously that we can’t have a good time.

Look, in the grand scheme of things, I don’t know if it’s right or wrong that we have mustache growing contests and embed random videos in our corporate blog, but it sure seems better than being just another faceless consulting company. That said, I’ve been wrong in the past, so what do you think? Handy poll included below for your convenience, but we also welcome comments with thoughts, feedback, and constructive criticism.

A Webcenter Interaction 10gR3 install and an Oracle 11g Database walk into a bar

2010-04-23T19:42:51Z

Error accessing database! Check your config files and verify that your database is correctly scripted.

It’s 2 AM, and these are not exactly the calm and soothing words you want to be seeing from PTSpy after a clean, fresh install of your WCI portal. It is frustrating, it is alarming, and above all else, did I mention it is 2 AM? The prospect of re-scripting a database by hand is just ever so slightly upsetting. So we look at little deeper at the Spy log:

Exception in thread “Main Thread” com.plumtree.openkernel.exceptions.OKSQLException: Error in function PTSession.Connect (iUserID == 2, SecurityProviderInfo == null): Error in function PTSessionInfo.GetCurrentCommunityMembership (): Exception: [plumtree][Oracle JDBC Driver][Oracle]ORA-00979: not a GROUP BY expression
SQL: ‘/* USER_COMMUNITY_MEMBERSHIPS:ANSI */ SELECT MAX(CS.ACCESSLEVEL) AS ACCESSLEVEL, C.NAME, C.ISLOCALIZED, C.OBJECTID, MC2.MEMBERSHIPTYPE FROM PTCOMMUNITIES C, PTCOMMSECURITY CS, PTVGROUPMEMBERSHIP GM, (SELECT MAX(MC.MEMBERSHIPTYPE) AS MEMBERSHIPTYPE, MC.COMMUNITYID FROM PTMYCOMMUNITIES MC, PTVGROUPMEMBERSHIP GM WHERE GM.GROUPID=MC.GROUPID AND GM.USERID=? GROUP BY MC.COMMUNITYID) MC2 WHERE GM.GROUPID=CS.GROUPID AND GM.USERID=? AND C.OBJECTID=CS.OBJECTID AND C.OBJECTID=MC2.COMMUNITYID GROUP BY C.NAME, C.ISLOCALIZED, C.OBJECTID, MC2.MEMBERSHIPTYPE ORDER BY LOWER(C.NAME) ASC’

Red Herring Alert: We wasted an hour and a half monkeying around with the Database SQL scripts and looking for an error there based on the PTSpy message above. That’s an hour and a half we’ll never get back.

Luckily, you will not have to go through this pain!

As it turns out, the real problem isn’t a mis-scripted database, but a compatibility problem between WCI 10gR3 and some Oracle 11g database default settings.

The fix is as follows:

To start let’s make a back up of our sensitive files just in case things go wrong (just copy them out to somewhere safe):
oracle\product\11.1.0\db_1\dbs\init.ora
oracle\product\11.1.0\db_1\database\iinitPLUM10.ora
Open sqlplus and connect as sysdba
SQL > alter system set “_optimizer_group_by_placement”=FALSE scope=spfile;
Stop the Oracle service corresponding to your database SID, it will be called OracleService<YOUR_SID>
Modify the file initPLUM10.ora and change ‘compatible = 10.2.0.0.0’ to ‘compatible = 11.0.0’
Start OracleService<YOUR_SID>
Reset IIS (use inetmgr to handle this using a pretty interface or open a command prompt and type: iisreset)
Clear your browser cache
Visit your portal home page.

For what it’s worth, the fix actually is documented somewhat by Oracle in the Release Notes for 10gR3 in the following statement:

* Note: When running WCI with Oracle 11g with the provided initPLUM10.ora file, make the following modification: Change compatible = 10.2.0.0.0 to compatible = 11.0.0. Also, to prevent problems with “group by” optimizations when using WCI with Oracle 11g you must add the following configuration to the bottom of your init$ORACLE_SID.ora file: _optimizer_group_by_placement=false.

The issue here, of course, is that if you did not read this single line addendum a little ways down in the release notes and instead you danced ahead in your portal install process the resulting error has nothing in the logs or the trace that would indicate that this is your issue.

Hopefully though, you are here now and have what you need to continue on. And this one author who will be paying just a little more attention to reviewing software release notes in the future, if for no other reason than to be able to go home before the birds start chirping again in the morning.

So here you are, you have just finished your install of Oracle WebCenter Interaction, 10g Release 3.

On top of that, you even opted to run the latest version of the shiny new database that Oracle has to offer, Oracle Database 11g.

And like a real man would – you read the instructions all along the way (Oracle WCI 10gR3 installation guide, WCI 10gR3 Release Notes), meticulously … at least for Oracle products.

C:\applications\oracle\product\11.1.0\db_1\database\iinitPLUM10.ora
C:\applications\oracle\product\11.1.0\db_1\dbs\init.ora

Function1 Welcomes Spheric to the Team

2010-04-15T22:21:29Z

Howdy all. This post is a bit late in coming, but exciting for me to share none-the-less. Function1 is pleased to announce its acquisition of Spheric Technologies. I could blabber on about what this means to us, our customers, and our loyal blog readers, but I’ll spare you all the pages of dialog and just give you a link to the press release (our first, so you know it’s a big deal) and copy/paste the announcement that’s live elsewhere on our site. To our our friends from Spheric….aloha, we’re happy to have you on board.

Please join us in welcoming Spheric to the Function1 family. This acquisition increases the value we deliver to customers and partners by leveraging the combined experience and expertise of both the Spheric and Function1 teams. We know many of you are wondering what this means, so please read on for more information.

Who is Function1?
Function1 is a premier solutions provider that takes a customer-centric approach towards developing usable, functional, and scalable solutions for enterprise customers. In addition to providing Oracle WebCenter professional services, Function1 offers packaged enterprise software, remote support, and hosting solutions to customers across the public and private sectors.

What does this mean for Spheric customers?
Current Spheric customers now have a bigger support system to lean on for help with their WebCenter deployments and development projects. Here at Function1, we have a great group of consultants and developers that together with the Spheric team will help you get the most value out of your portal investment.

What does this mean for current Function1 customers?
Current Function1 customers should look forward to new products and services that come from the combined team. We also have new service offerings and products related to Oracle’s Webcenter 11g release coming your way.

We’re thrilled to have Spheric as a part of the Function1 team. Keep up on the latest at Function1 by visiting our blog and following us on Twitter. Please contact us at info@function1.com with any questions – we look forward to hearing from you.

On Music and Using WCI as a Corporate Communication Tool

2010-04-09T00:05:14Z

Since the age of 12 or so, I’ve had an on-going love affair with music. You know that scene in Almost Famous where William is going through the record collection his sister left him and is just blown away by the realization that all this music exists?

Yeah, I had one of those moments, except it was with my parents’ record collection, and the hippest LP they had was Bob Dylan’s Greatest Hits Vol. II. Still and all, I can remember exactly the time lying on my bed flipping through an Uncanny X-men, and listening to Bob Dylan’s scratchy voice on scratchy old vinyl: the windows were open, it was spring. Who knows, it’s entirely possible that I’m over-romanticizing the memory, but after that random afternoon, I was all about music. I took up the piano and guitar, with varying degrees of success, and did my best to find new songs to put on mix tapes for friends.

But that was all years ago. It’s rare that I play much music anymore, and when I do it usually involves breaking out the acoustic guitar much to the dismay of my wife and dog. These days I mostly stick to listening as often as practical and acting mildly condescending whenever the topic of music comes up with someone who isn’t “into it”. I do still look for new music and share it with friends though. It’s just that now I post links on Facebook instead of making a mix tape (which is a shame, making quality mix tapes is a lost art). In any case, as I was telling Facebook friends about the awesomeness that is the Clash’s cover of “Police and Thieves” the other day, I got to thinking about the power of one-to-many broadcast messaging systems, and how you can use your portal to broadcast corporate communications out to your staff.

The Clash – Police And Thieves

For instance, I was recently working with a client who needed everyone in their organization (14,000+ people) to sign-off on some compliance documents. Getting 14,000 people to do anything is tough…so how would you approach the problem? If you’re like most people, you probably start by sending out an e-mail to all-employees@mycompany.com and ask them to fill out some PDF document, digitally sign it, and then return via e-mail. Shortly after hitting “send”, when 300 or so digitally signed PDFs hit your inbox, and you’re getting dangerously close to your mail quota, you likely get to thinking…”hmm….how am I going to track compliance, do I really want to manage this through e-mail and a big ol’ spreadsheet?” Then, after some more time passes, you realize that you really aren’t getting the response you’d hoped for, and that 70%+ of the workforce appears to have completely ignored your e-mail.

Time for plan B.

Enter your trusty intranet portal. Let’s assume that you have a relatively mature portal deployment that’s widely adopted as an employee intranet. The customer I was working with in this case, for instance, sets their portal as the default homepage for all employee browsers. As such, we know that the portal homepage gets very heavy traffic. Why not leverage all the eyes that are on that homepage to broadcast an important piece of corporate communication? And while we’re at it, why not just build a simple portlet that lets folks upload their signed documents, and enables management to run compliance reports? Sounds like a recipe for a pretty good portal use case. Let’s break down the components of the solution:

A reminder to non-compliant users that they need to upload a digitally signed document
A tool that allows users to upload completed documents
A tool that allows management to run compliance reports

In this post, I’m just going to dig into the details of implementing a user reminder, because it’s most relevant to the topic of using the Portal as a corporate communications tool. Before we get wrapped up in actual implementation details though, let’s take a minute to consider why the portal is a particularly well-suited solution to this problem:

We have a destination that most/all employees visit regularly, so we know our message will get in front of the target audience
If we wanted to, we could target our message as specific groups of users (only HR employees, or just the Sales Team, for example)
By using portlet preferences, we can make sure a reminder is only displayed to employees who have not complied with our message
Users are authenticated into the portal. This makes it easy for us to:
- Associate uploaded files with a particular employee
- Know who has viewed our message and did not comply with the request
- Know who has viewed our message and did comply with the request
- Know how many times users view the message on average before they comply

Okey Dokey, now we know what we need to build, and we’ve convinced the people that need convincing that it makes sense to use the portal for the project, let’s dive get down to the nitty gritty implementation details.

Implementing A Reminder For Non-compliant Users
Look, I hate pop-ups as much as the next guy, but they have their place. And using a pop-up to nag people who haven’t acted on our company wide request seems fair enough. So let’s look at implementing a modal pop-up window on the portal homepage. If you want to go all low-rent in your approach to the reminder, you could just add a little Javascript to your portal header like so and be done with it:

<script type=”text/javascript”>

var redirect = confirm(“Howdy. You might need to sign a document and upload. Go do it now?”);

if(redirect)

window.open(“http://some.portal.url”);

</script>

The result is something like this:

Not too pretty, and doesn’t give us a way to track who’s seen our message and who hasn’t. And besides, those old Javascript pop-ups are so 1998. How about a more elegant approach that uses a Javascript library? There are a ton of libraries you can use to do a modal screen overlay, pick your favorite. Here at Function1, we like extJs, so that’s what I’ll use for this example. Basically, we just want a pretty screen overlay that displays our message to users. Something like this:

Looking better, but we still don’t have a way to track who’s clicked through to comply with our message and who’s ignoring our request. And we can’t yet figure out how to display our pop-up to just non-compliant users. So how about if we wrap this pop-up in a portlet? Then we can use portlet preferences to keep track of who’s clicked through to comply, who’s ignoring the message, and who’s asked to be reminded later. Logically, our portlet should work as follows when figuring out when to display the pop-up message:

The trick now is to set a portlet preference based on how the user responds to our message. Keep in mind that our pop-up is all Javascript driven (i.e. client-side logic), and we need to get back to the server in order to set a preference. How do you figure we might accomplish such a feat? With our friend AJAX of course. ExtJS, and any other Javascript library worth it’s salt, provides a nice wrapper around XmlHttpRequest that makes it easy for us to do inline HTTP GETS/POSTS without having to refresh the entire portal page. So what if we just make an AJAX post to a JSP that sets a portlet preference for us based on the users response to our message? The logic would flow something like this:

OK, now we’re getting somewhere. We now have the mechanisms in place to:

Display a pretty pop-up for our message
Log user response so management can report on it later
Set a flag (via portlet preference) that determines when to display our pop-up

The only thing left to do at this point is to include some logic in our portlet to check the portlet preference, and decide when to show the pop-up reminder. Here’s a simplified version of the code that you might include in your portlet:

<%

//Get a handle to the portlet context so we can check preferences

IPortletContext portletContext = PortletContextFactory.createPortletContext(request, response);

IPortletRequest portletRequest = portletContext.getRequest();

IPortletResponse portletResponse = portletContext.getResponse();

boolean showPopup = false;

//read the preference and decide whether we should show pop-up or not

String showPopupString = portletRequest.getSettingValue(SettingType.Portlet, “showPopup”);

if(showPopupString != null && showPopupString.equalsIgnoreCase(“true”))

{

showPopup = true;

}

if(showPopup)

{

%>

<!– Javascript code to show pop-up here –>

<%

}

%>

And, viola, we’re done! According to WordPress, it only took me 1315 words to explain how and why you might want to use a pop-up in your portal. Let me now try to summarize a bit more succinctly, in the form of a Haiku:

Portal has many eyes.

Perfect for message broadcast,

and compliance stats.

I know, I’m a pretty good poet right? I’m embarrassed to tell you how long it took for me to be sure that “compliance” was a three syllable word, so I won’t. In any case, if you’re interested in tinkering around with this example, first go download extJS 3.2.0 and then grab the portlet code here:

popupPortlet.jsp – This is your portlet that displays the pop-up message and contains display logic. Set your WebService to point at this file.
ignore.jsp – This page just sets a portlet preference in response to an AJAX post
later.jsp – This page just sets a portlet preference in response to an AJAX post
myApp.jsp – This is a place-holder page that sets a portlet preference and would presumably contain the application you want users to step through.

And that’s just about a wrap. One more thing though. Remember that customer I told you about earlier who had to get 14,000+ employees to comply with a company wide request? Well, they started out managing the request process via e-mail, and after many weeks of repeated e-mails, their compliance was at about 15% of the workforce. After we implemented the portal pop-up reminder, compliance jumped to 60%+ within 4 days. Not too shabby.

…

Bah, I can’t resist putting up one more MP3 for you guys. I’ve never been a huge rap fan, but you can’t deny good hip hop. Enjoy this mash-up that let’s me pretend to have some indie rock cred while blasting the song that made Biggie a household name:

Notorious B.I.G vs. The XX – Juicy

Terms of Endearment – Part 1: Understanding High Availability

2010-04-01T16:48:48Z

Hello Everybody…so this is my first post of a multi-part series of topics covering terms that are often misused or generally confusing. There are a number of topics I want to cover at a general 101 level before looking at each in more detail during a 202 series.

So without further a dieu, those of you that have had the pleasure of working with us may already know that for some reason people confuse Vasanth for me, and vice versa. While we are both awesome people, I assure you we are in fact different people with different parents that aren’t even related to each other. Never the less, some people may still expect me to respond if you call me by his name. Clearly this is some funky use of logic. Interestingly enough, that same logic seems prevalent with regard to technology terms. Say one word, mean another. This post is about the difference between Redundant, and Highly Available. Much like Vasanth and I, these terms are quite different from each other.

First lets talk about “redundancy”. Redundancy almost always refers to the hardware serving up your application and at times can extend to the infrastructure surrounding it. Redundancy is a term that is often used in explaining High Availability, but just because something is redundant does not mean that it is Highly Available. The definition of redundancy is a subset of the definition for High Availability. In context, redundancy would imply that the physical hardware hosting your application includes multiple components of each type. An example of hardware redundancy for your average everyday web server would be 2 processors, 4 heat syncs, 8 system fans, 2 power inlets, 2 network cards, 2 hard drives, etc. Redundancy has applicability on various scales. Power redundancy, for example, refers to both the power supply to your server as well as the supply to the rack that holds your server. Network redundancy refers to the switch your server is attached to as well as the network interface(s) on your server itself…and so on. There are nearly a dozen other areas to discuss but I will save that deep drive for a future post.

So what does this have to do with High Availability and what does “highly available” mean? Simply put, an application that is highly available (a.k.a. HA) is one that is available to users 24 hours a day / 7 days a week with very little (if any) down time for maintenance. This implies that the hardware and infrastructure surrounding your application is redundant. That’s the simple definition. In greater detail though, the term highly available means your application must be available for 90% to six nines. High Availability is usually measured as a percentage of up-time for a given system. It is calculated throughout the year and marked at year end. Each organization can make the determination as to what is and is not an acceptable percentage of up-time. Six nines is the extreme of HA. Six nines availability is exactly what it sounds like: 99.9999 % up-time. That means 2.59 seconds of downtime per month or 31.5 seconds a year. There is however a bit of a grey zone around what qualifies as downtime. Downtime refers to a period where the application is unavailable to users. However planned downtime has a different weight than unplanned downtime. Planned downtime is generally introduced to the environment for OS patches, software upgrades, and alike. Unplanned downtime is due to logical fault like an application crash, or physical fault like a damaged power supply.

HA implementations can vary greatly depending on what type of application you are deploying and it’s dependencies. Often people are mislead to believe that a system design like this has lots of servers, and extra network switches, and multiple firewalls, and all sorts of other crap … but in reality an HA system is allot simpler than that. The larger the design the more complex it gets, because each component added introduces a new potential failure. Some of the most highly available systems are simple. An example of a simple design would be a pair of single rack mount servers physically located in different locations each with redundant internal hardware such as mirrored drives, multiple NICs, multiple power supplies (all engaged), and two or more CPU’s that are running processes independently. In a nutshell, that is HA for you and we’re barely skimming the surface of this topic. There are a number of other areas that are closely linked to this concept such as fault tolerance and reliability, but in an effort to keep this first post as the 101 to this series, I will go into more detail on that in some future blog post.

Okay, so now that we’ve learned our terms, lets use it in a sentence: I know my applications are highly available because my up time is greater than 90%, my hardware is internally redundant, and it is physically located in separate locations.

I hope you’ve found this post informative, see ya next time!

WCI Search Overload Syndrome

2010-03-26T16:32:19Z

Recently, one of our customers asked us to help them improve the search experience for their 6.5 portal users, and before I could even think about the problem, the first thing that popped in my mind was this:

This commercial is so telling of the search-overloaded world we live in, where we are constantly searching for information since the results are so handy. On some days, I even feel like Nicolas Carr and want to throw out my laptop! Well … may be after a few more searches.

After “researching” this syndrome on the internet, I came back to focus on this customer’s issue. In general, their portal users were finding that relevant content was not showing up at all or was showing up further down in the list of search results than expected. While most of the issues were solved by repairing the search index and tuning other search features, there was still one quirky search scenario which was still returning unexpected results to their users when they performed a multi-word search.

For instance, if a portal user searched for ‘Lincoln Abraham’ instead of ‘Abraham Lincoln’, the search results returned something like this:

1.      Biography of Lincoln, Abraham
2.     16th President: Lincoln, Abraham
…
12.   Abraham Lincoln

Most users of Google and other search engines would expect result #12 to show up first. In WCI, we could set a best bet to force #12 to show up first in the search results, but there were too many cases like this for this customer. The answer to this problem lies in the algorithm used by the portal for the default search mode. I recalled that I had come across this algorithm in a document some years ago but couldn’t remember where. Ironically, it could not be found by searching the internet, but on my backup drive. It was actually in the Search Server section of the good ole Plumtree 4.5WS Deployment Guide, which I have no idea why I still have saved but just thankful I did. I’m not sure if or where this is in the current documentation, but I’ll save you some trouble by providing the section here: Plumtree Search Engine

Before I jump into showing how to fix the problem, let’s quickly review the seemingly simple search functionality of the portal. I’m sure you’re saying: “What’s there to review? … I know how to search … I just type in some search terms and click the search button”. While that’s true, and that’s what most of your portal users are doing, there are more options available there to narrow the search results, which are definitely useful!

Including the default mode that most users use, the portal search actually supports the following 3 modes for queries:

Default Mode – list of strings without any reserved characters/words from the other 2 modes listed below.
Internet Mode – using include and exclude operators noted with ‘+’ and ‘-’ operators. The ‘+’ operator requires that word in the search result, and the ‘-’ operator returns results without the corresponding term.
Query Operator Mode – uses a set of reserved search keywords to create complex query combinations. Here’s a list of the reserved words :
- AND , OR, NOT
- NEAR , NEAR/ – use to find words that are near each other in a phrase. Default proximity is 25 words, but NEAR/10 can be used to check 10 words apart instead
- <ORDER> -
- <WORD> - find exact match on word with case-sensitivity
- <ALL>, <ANY>

If the search server gets a query string which contains a reserved term for the latter 2 modes, then it uses that corresponding mode automatically to conduct the search. Otherwise, if there’s no reserved word, then it uses the default mode. All modes support the use of wildcard character (*) and the use of double quotes for phrase-search (i.e., “President Abraham Lincoln”). Now that we’ve covered the query operator mode, it’ll make it easier to understand the algorithm for the default mode. To illustrate the algorithm for this default mode, we’ll use the same example as before. When the user searched for “Lincoln Abraham”, the search server interprets this as a default mode query since there are no operators in the query string. Here’s the actual 3-part algorithm that the search server uses to perform the default-mode query:

NOTE: The first part of the query ranks higher, so the results for part 1 will show up in the search results before the results for part 2.

So, in other words, the default search mode is just a fixed query operator mode search under the covers. With this revelation about the default search behavior, the question still lingers … How do we fix this customer’s problem? Well, after some testing, it turns out that default search was causing problems for this customer due to the <ORDER> keyword in part 2 of the default mode algorithm. And, there are a couple of solutions for fixing this:

Option 1:   Upgrade to 10gR3 , which doesn’t seem to have this problem
Option 2:   Write a custom Programmable Event Interface that modifies the query string (see PEI in the WCI UI Customization Guide)
Option 3:   Replace the default search in the layout with custom search banner code

Both options 2 & 3 will need to convert the query from a default mode one to a custom, query operator mode before submitting to the search server. The custom query algorithm will be the same as the default mode without the <ORDER> operator. For this customer using WCI 6.5, option 3 actually gives us the simplest and least intrusive way to fix this issue. We did not end up implementing the solution for the customer as they chose to do this on their own. But, here’s a quick stab at how the code would be put together:

In the base page layout file, we find the existing search UI and replace it with HTML like this:

Also, the psuedo-code for the javascript method mentioned above (customizeSearchString) would be something like:

1) Get the current value of the fake_q_id text field

2) Token-ize the text string entered by the user in the fake_q_id field

3) Don’t convert if there’s only 1 token (single-word query) or if any of the tokens are a reserved word for search

3) Using the tokens, construct a replacement query that doesn’t use the <ORDER> keyword.

4) Set the constructed query to the hidden q_id field

And that’s it! This javascript function essentially performs the conversion of the default mode query to the custom one we needed. As for the customer we had supported, they have implemented a similar solution and the pilot users have been pleased! At the end of the day, it’s all about making the portal as intuitive and as productive as possible for your users. And now that we understand the portal’s search functionality better, it should make it easier for us to do exactly that.

Happy searching :)

Everything Maven Part 2 – Twitter Portlet

2010-03-18T21:25:26Z

A few weeks ago I wrote about reasons why you might want to invest some time in learning and using maven: “Everything maven Part I”. Now it’s time to roll up our sleeves and get down into guts of maven. In this article, I’ll describe how you can use Maven to build and deploy a WCI portlet using the IDK. The portlet will connect to Twitter and return all your followers’ latest tweets.

Step 1: Install Maven

Here’s a link to the official maven2 installation guide. Here are the steps I used to install on my windows laptop:

Download and install the Sun Java JDK 6 update 18. Here’s the link: http://java.sun.com/javase/downloads/widget/jdk6.jsp.
Download the latest maven installation zip file from here http://maven.apache.org/download.html
Extract the zip and move ‘apache-maven-2.2.1′ into c:/Program Files/Apache Software Foundation/apache-maven-2.2.1
Create a windows environment variable named “M2_HOME” and pointed it to the maven installation directory: M2_HOME=c:\Program Files\Apache Software
Foundation\apache-maven-2.2.1
Create a windows environment variable named “M2_REPO” and point it to “C:\Documents And Settings\dave.m2\repository” (this will make more sense after you read the section below on repositories).
Make sure that the “JAVA_HOME” environment variable is set. For example, the JAVA_HOME environment variable on my windows laptop is set to C:\Program Files\Java\jdk1.6.0_18
Add the mvn command to the windows “PATH” environment variable like so: PATH=…;%M2_HOME%\bin
Finally, Test to ensure maven was installed correctly. Open a Windows DOS Command Prompt, and type “mvn -version”. If you see the following you should be good to go:
C:\Users\Dave>mvn -version Apache Maven 2.2.1 (r801777; 2009-08-06 15:16:01-0400) Java version: 1.6.0_18 Java home: c:\Program Files (x86)\Java\jdk1.6.0_18\jre Default locale: en_US, platform encoding: Cp1252 OS name: "windows 7" version: "6.1" arch: "x86" Family: "windows"

Maven is installed. Let’s take a break, here’s an airplane landing with one wing:

Step 1b: Get the code

Here’s a Zip containing the complete portlet code: function1-twitter-portlet. Feel free to refer to this as I describe the steps to create a maven portlet project below. This zip contains a fully functionaly maven project. Extract the zip and open the pom.xml file. Add your twitter credentials to the file, and then open a command prompt. Change directories to wherever you extracted the zip and then run the following command:

mvn install

If maven is installed correctly, and your twitter credentials are valid, this command should download required jars from maven, run tests to make sure it can connect to twitter, and build a deploy-able portlet war file under directory named ‘target’!

Step 2: Create Basic pom.xml

Now that Maven is installed, we need to create an empty Maven Project. A the bare minimum, a Maven Project is nothing more than a folder that contains a single file named ‘pom.xml’. In Maven terminology, “POM” stands for “Project Object Model”. Maven will look for a pom.xml file to discover everything it needs to know about our Twitter Portlet Java Project. Once you’ve used Maven for a few different projects, this will become second nature…I promise. Eventually you’ll have a template pom.xml that you can simply copy into a new directory each time you’d like to create a new maven project.

First, we’ll create a new directory named ‘twitter-portlet’ (or just follow along by browsing the code from the zip file). Next, create a new pom.xml file and copy in the following

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.function1</groupId>
  <artifactId>wci-twitter-portlet</artifactId>
  <packaging>war</packaging>
  <version>0.1</version>

  <name>WebCenter Interation Twitter Portlet</name>
</project>

So far, we haven’t done anything more than describe our Twitter Portlet Project to Maven. Here’s a short overview of each line:

The “modelVersion” is required by all Maven2 Projects and always has the value 4.0.0.
The “groupidId” tells Maven that this project belongs in the “com.function1″ group. The best practice is to use your company’s name for the groupid, but it can be whatever you choose. Multiple projects (multiple pom.xml files) can all share the same group id.
The “artifactId” defines the name of our specific project: wci-twitter-portlet.
“packaging” tells maven what type of project we’re building. 99% of the time, this is either going to be “war” or “jar”. In this case, of course, we want a war to deploy as a remote portlet.
The “version” tag defines what version of the wci-twitter-portlet we’re working on. In this case, its the first version ever so I chose to call it version “0.1″.
The “name” is just a human readable name. Maven will use this when generating documentation.

Step 3: Directory Structure

Maven is very particular about the directory structure of a project. If this is the first time you’ve used Maven, this might make you feel a bit trapped. When I first started with Maven I tried to fight it by using my own directory structure. Over time, I’ve come to see the light. Trust me, just go with it. I’ve realized that the directory structure used by Maven is a proven best practice and in the end it really makes things consistent and easy. Here’s what the basic directory structure looks like:

Create a directory named src/main/java. This is where we will put custom java code.
Next create a directory named src/test/java. This is where unit test code goes.
Now create a directory named src/main/resources. Here’s were all your configuration files go. For example: DTD, .properties files, xml files (such as log4j.xml), etc.
Finally, create a directory named src/main/webapps. Here’s where your jsp’s, and WEB-INF/web.xml file, and any static stuff like images, and javascript.

Ok, directory structure is done, Time for a break! I think the things that this guy builds are freaking cool:

Step 4: Generate an Eclipse Project

By this point, we’ve given maven enough information to be able to create an Eclipse project for us. switch to your command prompt and run the following command:

mvn eclipse:eclipse

You should see a message “BUILD SUCCESFUL”. Open Eclipse and choose “File -> Import -> Existing Projects into Workspace”. Choose the twitter-portlet directory and import the wci-twitter-portlet eclipse project (notice Maven used the artifactid for the name of the eclipse project).

Step 5: IDK and DEPENDENCIES

Our Twitter Portlet will need to use the IDK to get the current username. We’ll need to depend on the WCI IDK. So, the next step is to download the IDK (if you haven’t yet). Once it’s downloaded, extract the installation zip and then find the WEB-INF directory under idk/10.3.0/devkit. Copy the WEB-INF folder into the wci-twitter-portlet directory into src/main/webapp/.

Now, delete the lib directory! Yep, you heard me. Just like taking off a bandaid, rip that lib directory out of there. After reading this article, you’ll never have to copy the idk lib directory again (ok, except maybe when they upgrade to 11g). Now it’s time to tell Maven to manage these jars for us.

Maven Repositories

Which leads us to the concept of Maven “repositories”. Maven looks for jar dependencies inside repositories. The Maven Book does a much better job at describing the concept than I can, but basically, a repository is simply directory structure containing maven artifacts.

By default, Maven looks first for dependencies here: http://repo1.maven.org/maven2/

Thanks to convention over configuration, everything inside the repository is organized using the following structure:

groupId/artifactId/version/artifactId-version.ext

For example, log4j, version 1.2.9 can be found here:

http://repo1.maven.org/maven2/log4j/log4j/1.2.9/.

In this case, groupid and artifactid are both set to ‘log4j’. Here’s another example, ‘axis’ can be found here:

http://repo2.maven.org/maven2/org/apache/axis/axis/1.4/

There are several tools available to help search the maven repository for stuff. There’s a Maven-Eclipse plugin that allows you to search and automatically add dedpendencies to a maven project. There’s also a website that allows you to search the maven repository: http://mvnrepository.com/. For example, when you do a search for ‘log4j’ this site will show you all available versions. If you drill down into version 1.2.15, you’ll see the following:

<dependency>
    <groupId>log4j</groupId>
    <artifactId>log4j</artifactId>
    <version>1.2.15</version>
</dependency>

I know, I know, xml makes my eyes glaze over too. To be honest, I’m not crazy about the fact that maven relies 100% on xml. When you write java code, you gotta deal with xml, what can I say?

So, if we added this snippet of xml to or pom.xml, maven will know that our project uses log4j. In fact, maven will be so nice as to find and download the appropriate log4j jar and make it available on the classpath of our project.

Maven is smart – as smart as they come. Instead of downloading jars every time you do a build, it only downloads the jars once. And instead of downloading the same jars into every maven project, it downloads them into a directory under your operating system user’s home directory. For example, on windows, this is C:\Documents and Settings\Dave\.m2\repository. On mac, this is /Users/Dave/.m2/repository.

Ok, that was a lot. Here’s a quick summary: The Maven repository located at http://repo2.maven.org is a remote repository. When you tell maven that you need log4j by including the xml snippet above, it will first check your local maven repository (which is under your os user’s home directory). If it can’t find the jar, it will then check http://repo1.maven.org/maven2/.

Dealing with IDK Jars

So, the problem is that none of the idk jars are available in either of those places. To solve this, we’ll need to copy the idk jars into a local maven repository of some sort.

The first step in doing so, is to re-arrange the jars inside the idk’s lib directory to match the directory structure that maven expects. Like so:

Next, we need to tell Maven to look at our new local directory repository in addition to the default repo at http://repo2.maven.org/maven2. Here’s the snippet of xml inside pom.xml that does so:

<repositories>
  <repository>
    <id>thirdparty</id>

    <name>Thirdparty Jars</name>
    <url>file://${basedir}/thirdparty/m2/repository</url>
    <layout>default</layout>
  </repository>

</repositories>

Now that the idk dependencies are available, lets add them to our project. We’re going to add a snippet of xml for each jar. Here’s a few examples to give you an idea:

 <dependencies>
   <dependency>

     <groupId>com.oracle.idk</groupId>
     <artifactId>activation</artifactId>
     <version>10.3</version>
  </dependency>

  <dependency>
    <groupId>com.oracle.idk</groupId>
    <artifactId>axis</artifactId>
    <version>10.3</version>

  </dependency>
  <dependency>
    <groupId>com.oracle.idk</groupId>
    <artifactId>commons-discovery</artifactId>
    <version>10.3</version>

  </dependency>
... more depenedencies ...
</dependencies>

Phwew! We’re getting to the end…before we do, here’s a old spice commercial that cracks me up:

Step 6: Write the Code!

At this point, we have everything to need to create a portlet that uses the IDK. Please take a look inside the zip to see the code. But why stop there? Let’s add some twitter! It just so happens that there’s a library called twitter4j available via maven:

    <dependency>
      <groupId>org.twitter4j</groupId>
      <artifactId>twitter4j-core</artifactId>
      <version>2.1.0</version>

    </dependency>

Run mvn eclipse:eclipse again to update your eclipse project with the new twitter4j jar. (If Eclipse is open while you run this command, remember to do a “refresh” in Eclipse so that Eclipse see’s the new jar)

Now, we can add some code to our portlet that uses twitter4j to retrieve the latest status from each twitter friend. Here’s the relevant snippet. Feel free to peruse the full source int he zip file:

 public List<String> getFriendsMostRecentTweet() {
    List<String> tweets = new ArrayList<String>();
    String senderID = getTwitterLogin();
    String senderPassword = getTwitterPassword(); 

    Twitter twitter = new TwitterFactory().getInstance(senderID,
            senderPassword);
    PagableResponseList<User> statuses;
    try {
        statuses = twitter.getFriendsStatuses();
        for (User nextTwitterUser : statuses) {
            tweets.add(nextTwitterUser.getName() + ": "
                    + nextTwitterUser.getStatusText());
        }
    } catch (TwitterException e) {
        System.out.println("Unable to retrieve tweets");
        return null;
    }
    return tweets;
}

Finally: Running the command “mvn install” will create a war file named wci-twitter-portlet-0.1.war under the target directory. Deploy that war to tomcat (or weblogic or whatever you’re using as a remote java portal server). Create the remote server, webservice, and portlet objects inside the WCI Portlet and marvel at the twitter messages!

“Just When I Thought I Was Out, They Pull Me Back In”: A Belated Welcome To Dave Paroulek

2010-03-11T20:25:23Z

Look, I think we can all agree that The Godfather III sucked, and was a black mark on an otherwise amazing trilogy. The one (and probably only) highlight of the movie was Pacino giving us the sound-bite, “Just when I thought I was out, The pull me back in!”. And so we welcome Dave Paroulek to the Function1 team.

As you may or may not know, the WCI professional services world isn’t all that big. Your options for finding quality consultants are pretty much either: Oracle (for an un-godly hourly tab), us, or a couple of other firms with WCI expertise. The small resource pool of qualified WCI consultants means that when we’re looking to bring new folks on board, we start by talking to people we’ve worked with in the past (usually at Plumtree/BEA/Oracle). These folks are a know quantity, and we know they’ll represent Function1 well and make our customers happy. Such is the case with Dave. Dave is another long-time Plumtree->BEA->Oracle guy who took a little different career path than the rest of the Plumtree alumni at Function1. He put his time in working for Plumtree Public Sector professional services, and then moved onto the BEA Services Engineering team. In this capacity, Dave was focused on designing, developing, and delivering WCI add-on software directly to customers. But his development experience doesn’t stop there. About a year and a half or so ago, Dave left the mothership and took a full-time gig doing software development (specifically, playing around with GWT).

And now he’s back in the fold. Absence makes the heart grow fonder I suppose, and when we talked to Dave about coming on board at Function1, he was all about it. Dave comes to Function1 helping us codify our software development practices and expertise, and continue on our quest to ~~take over the world~~, err, provide world class services and software to customers everywhere. We’re tickled pink to have Dave with us: be on the lookout for more kick-ass blog posts on software development tips and best practices from him soon.