Showing posts with label Troubleshooting. Show all posts
Showing posts with label Troubleshooting. Show all posts

Common Problems encountered in WebSphere environment

WebSphere Environment Setup: 

The below mentioned issues are most commonly found if you have the Split web server and app server configuration, which means you have web server running on different physical box serving static contents and appserver running on different physical box serving dynamic content. 

1. Web server plug-in updation

Due to the split webserver and appserver setup, plugin-cfg.xml file from appserver needs to regenerated/updated on web server for the changes to reflect. 

You can suspect the need of plugin updation whenever the application is not working after the first deployment, new features involving the servlets does not work. Also in the webpage you get message saying 404 path not found. 

Below mentioned are few of the instances you need plugin-cfg.xml propagation:

a. When you deploy any new application. 
b. When you add new virtual host entry in the appserver plugin file. 
c. When application incorporates new servlets in the code.

2. Time mismatch in Cached JSP page and the Deployed JSP page. 

You can suspect this when user complains about the changes not getting updated, this is due to the fact that JSP file deployed newly has older time stamp then the JSP file in the cached path /global/site/vendor/WebSphere/AppServer/temp, due to which the latest JSP file never get compiled. 

Solution is touch the specific JSP page and redeploy the application.

3. File upload problem, permissions issues with the existing id

Since web server serves the static content whenever the new file is uploaded to the appserver, the same needs to be propagated to the web server. 

Below mentioned are the requirements to have file upload utility function properly. 

a. File upload directories should have the full permission for the file upload user(Mostly this will be websphere admin id) 
b. Setup SCP without prompting for the password for the application id( It will be generally websphere admin id) between websever and appserver. 

4. Database index corruption.

You can suspect this when database intensive application is suddenly very slow, this may be due to the fact that database table index might have been corrupted. 

5. Copy the WAR/EAR file in the BIN format

It is always good to ftp the WAR/EAR file in the BIN format for deploying the application. 

6. File serving variable should be false in the WAS environment

In split environment web server is serving the static contents, so in order to maintain that file serving variable should be false. You face problem with the static entries not displaying. 

#/clocal/www/servlets/webpub.ear/webpub.war/WEB-INF/ibm-web-ext.xmi 
“FileServingEnabled="false

Also Alias created in the webserver config may not be proper. 

7. Corruption in XML/INI files of the application

You can hint at XML properties file corruption, when you see that after deployment the application is not throwing login page itself. It may be missing variable in the XML file or the variable mentioned are not in single line. Also you can open the XML file in the browser, if it is corrupted then it will fail to open. 

8. Virtual Host not found error when you are accessing application from the browser.

Virtual Host not defined in the web sphere configuration, meaning VH with which you deployed application was never created in WAS5.

Application might have wrong VH name in the /WEB-INF/ibm-web-ext.xmi file.

DNS name used for the application may not have been mentioned in the plugin file under specific VH. 

Example: <VirtualHostGroup Name="wcm_elearning"> 

<VirtualHost name="www.abc.services.com:80"> 
</VirtualHostGroup> 

9. Java out of memory exception.

Start with the application debug first, if application is having memory leaks, you can detect the same using the JPROBE tool (Run time analyses tool) and also Jtest tool with does static analyses of the code.

The last option should be to increase the JVM memory (HEAP SIZE), if you are absolutely convinced that the application needs so much memory for the proper functioning.

This could also be due to LDAP connection not closed after the use. 

10. Connection pool running out of connections.

Start with the application, first option is to use JTEST tool to find weither all the opened connections are closed. If tool is not available check the code manually.

Run through the JPROBE tool to detect any over head functions, could this also be related to logic of closing the connection in the finally method, for some reason finally method does not get executed.

WebSphere Application Server troubleshooting tools

Troubleshooting a problem is an integral part of developer's profile and troubleshooting a problem without proper set of tools could be a very painful process. Below is the tech note from IBM which talks about the  troubleshooting tools that can be used to troubleshoot applications deployed on WebSphere Application Server.

WebSphere Application Server Troubleshooting Tools

Java Troubleshooting Tools

Cross-product Troubleshooting Tools

Error : java.exe has encountered a problem and about to close

This kind of error occurs usually when trying to publish a store in WCS. While trying to publish a store using Commerce Admin Console, java.exe terminates by showing a windows error message.

Solution:

The cause of this problem is due to Java JIT (Just In Time) compiler interference in compilation of java files. So disable JIT after login to WebSphere Administrative console.

  • Go to http://localhost:9060/ibm/console
  • Click login.
  • Navigate to Servers>Application servers>server1
  • In the Configuration tab, under Server Infrastructure, expand Java and Process Management.
  • Select Process definition.
Click on Java Virtual Machine link under Additional properties on the right panel.
Now the bread crumb shows ‘Application servers > server1 > Process Definition > Java Virtual Machine’

  • Select the checkbox ‘Disable JIT’.
  • Save the changes.
  • Restart the server.
  • Try to publish the store again.  

WebSphere Portal server is in starting status always

Issue Faced: WebSphere server status is always "Starting" after enabling the Global Security by double clicking on server.

Solution: Follow the below steps for resolving this issue.

Open soap.client.props in base_v6\profiles\default\properties folder and add the user-id and password fields as shown below. The user-id/password must be the same as the one given on “custom user registry” page.

com.ibm.SOAP.loginUserid=admin
com.ibm.SOAP.loginPassword=admin 

WCM authoring portlet and page not getting displayed after portal migration

If you are planning to migrate to WebSphere Portal 7.0 or higher, it is very likely that you may not be able to view the WCM out of box page or the WCM Authoring portlet. In some cases the user interface of WCM authoring portlet might not work as expected and you might end up getting the following exception in logs.

IWKAP0009E: Servlet not enabled

If you see the above exception, it's time to re-enable some of the WCM features manually post migration because he migration process sometimes disable some new out-of-the-box features if the feature did not exist in the previous version or if it was updated. To fix this issue, you have to re-enable the affected features manually after migration.

Navigate to folder \wp_profile_root\Config Engine directory and run the following config engine task.

Note: For clustered environment, run this task on the primary node of the server.


Windows:
ConfigEngine.bat install-wcm-hidden-authoring-page -DWasPassword=<password>

UNIX:
./ConfigEngine.sh install-wcm-hidden-authoring-page -DWasPassword=<password>

Few points to remember:


  • Make sure that the page on which the WCM authoring portlet is installed uses a theme with client-side rendering such as the Page Builder theme.
  • Deploy the hidden Portal page on which the reserved authoring portlet is by default installed. 

The Web Content Management applet failed to load or is unavailable

Issue Faced : WCM throws an error message while creating a new Authoring Template something like


The Web Content Management applet failed to load or is unavailable. Please contact your administrator immediately.

Solution : The solution is to check the installed JRE version on your machine :

The Web Content Management Authoring Portlet requires one of the following JRE's to be installed otherwise it will throw the above exception: 

Few websphere portal related issues and their solutions

 Dojo not found on portal page. If you are facing any issue related to dojo_en.js script on portal page, try the below solution.

Solution:

Set the profile to "Full" under page properties. Refer the screenshot.

Dojo


Issue 2:

Invalid file system control detected.

FileMultiRead E com.ibm.icm.jcr.FileMultiReadBinaryValue extract javax.jcr.RepositoryException: Invalid file system control data detected
javax.jcr.RepositoryException: Invalid file system control data detected
at com.ibm.icm.jcr.FileMultiReadBinaryValue.getFileOutputStream(FileMultiReadBinaryValue.java:100)
at com.ibm.icm.jcr.FileMultiReadBinaryValue.store(FileMultiReadBinaryValue.java:234)
at com.ibm.icm.jcr.FileMultiReadBinaryValue.extract(FileMultiReadBinaryValue.java:340)

Solution:

If you saw the above exception in the logs that means the directory where Portal write temp files for binary values is corrupted. Here is the location of the directory.
/wp_profile/PortalServer/jcr/BinaryValues
Empty the directory and try again. It should resolve the issue.

Issue 3: 
Error encountered while running the XMLaccess script.

<request xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" build="wp601_194" type="update" version="6.0.1.0" xsi:noNamespaceSchemaLocation="PortalConfig_6.0.1.xsd">
<status element="web-app 1_DECEHHG30OO9702RTP0TSV1000" result="failed">
<message id="EJPPH0007E">com.ibm.wps.pe.mgr.exceptions.InstanceException: EJPPH0007E: Instantiation of the administration module for Portal context /wps failed.</message>
<message id="EJPPH0010E">com.ibm.wps.pe.mgr.exceptions.InitException: EJPPH0010E: Failed to retrieve and initialize runtime variables from Application Server.</message>
<message>com.ibm.websphere.management.exception.ConnectorException: ADMC0053E: The system cannot create a SOAP connector to connect to host localhost at port 10033 with SOAP connector security enabled.</message>
</status>
</request>

Solution :

Check if your WAS is working properly and the SOAP port is correctly defined. Check your WAS SOAP port again and assure it is set correctly in your Portal configuration.

Difference between Friendly URL’s and URL mappings

In this article, I am going to talk about friendly URLs, URL mappings and difference between them.

Friendly URLs


Friendly URLs are human readable URL prefixes generated based on the content node hierarchy. The URL prefix for content sub node will contain the prefix assigned to the super and all other nodes above it. Friendly URLs are added to a page through page properties in Admin UI or we can add them through XML access too.



By default, Friendly URLs are disabled in websphere portal, to enable them

1. Open WAS admin console.
2. Navigate to Resources -> Resource environment -> Resource environment providers.
3. Search WP_ConfigService and click custom properties.
4. Click “NEW” and add “friendly.enabled” in name and “true ” for the value.
5. If it already exists change “friendly.enabled” to true

The URL assigned is stored in the page level metadata as com.ibm.portal.friendly.name.

URL Mappings

URL mappings are used to define the human readable URL prefixes to the content in the portal which will serve as entry points onto the portal. However, these URL mappings will not be preserved upon navigating to other content node. We can define URL mappings through Admin UI and XML access. Portal does not provide any API to construct URL mappings programatically. URL mappings are preferred when we need a URL to link up between two different portals. This will allow us to maintain a constant URL even with change in content node.



Few Important Points


1. We need to maintain URL mappings for each entry point and they will not be valid for the next navigation state.
2. We need to define friendly URLs for all the pages or nodes in the hierarchy before assigning it to sub pages. It cannot resolve a friendly URL unless and until all pages are properly defined with respective URLs.
3. For friendly URLs, URl assigned is stored in page level metadata as com.ibm.portal.friendly.name.

How to take a heapdump and javacore of running Portal Server instance?

Taking a heap dump can help you to diagnose the hung threads on a Portal system. A heap dump is nothing but a dump of memory of the Java heap. Similarly a java core will provides you some insights into the multiple processing running inside of the heap at a given point of time.

Note: Java cores should be taken two minutes apart in order determine if processes are hanging across intervals.

Depending on the version of Portal or WAS installed, the heap dumps and java cores are placed in different locations on the filesystem.


For Portal 5.1, the files are located in folder /WebSphere/AppServer /
For Portal 6.x, the files are located in  folder /IBM/WebSphere/AppServer/profiles/wp_profile/

The naming convention of these files is something like heapdump.20130318.151306.1168.phd and
javacore.20130305.085618.4412.txt

Below are the steps for creating a heap dump using simple command:

1. Find out the PID of the Portal server by typing the command 'ps -ef|grep java'.
2. Type 'kill -3 ' on the prompt and wait for some time.
3. Verify that the heap dump and javacore are created in the targeted directory.
4. Wait for couple of minutes, and then again run 'kill -3 ' command.


or

Another simple way of creating a heap dump in windows environment is to :


Heapdump
Heapdump
  • Right click on task manager. 
  • Look for the running JVM process.
  • Right click again and click End process to kill the selected process.

Step by Step WebSphere Portal Server troubleshooting guide

Have you ever been in a situation where in you are working on some important project deliverable(s) and suddenly you realize that your Portal Server has stopped working.

What would you do in that situation? 

You have couple of options to try here : Either troubleshoots the problem and try to found out some solution or easiest thing to do is to reinstall the portal server again and start all over again.


This article talks about the scenarios that you might encounter while working in the WebSphere Portal environment:

  • Portal JVM Crash : WebSphere Portal is considered to have crashed if its process identifier (PID) is disappeared from the list of processes or if the PID is changed over time without intentional stopping of the server, such as stopServer command being issued.
  • Portal Run-time failure : This is any failure that may occur during general operation and administration of WebSphere Portal Server.
  • Portal Server Start time failure : This is any failure that causes the Portal not to start or can allowed to start but is no longer accessible via URL.
  • Portal Server Operating System failure : This is any failure that is caused by the Operating System, for example: the accidental removal of the WebSphere folder.


Portal JVM Crash


The following diagnostics logs and files are essential to troubleshooting a crash:

Java core : This file is generated when the JVM terminates unexpectedly. It is a text file that contains information about the JVM and Java application captured at some point during execution.

Core Dump or Heap Dump : This file contains a complete dump of your computer’s memory, and therefore, it can grow large.

Check his article here to see how to create heap dump and javacore files.

The following are the steps to determine this issue:


  • Search for the Javacore files (/IBM/WebSphere/AppServer/profiles/wp_profile). If there is a Javacore present, analyze that file and find any Signal 11, since this indicates the application server crash.
  • If there is no Javacore, the issue may be solved by upgrading the IBM Java SDK with the latest service release because the JVM is probably hung.

Portal Run Time Failure


This could include, but is not limited to, the following:
  • Unable to login
  • Administration tab unavailable
  • Portlet deployment failures
  • Portlet unavailable
  • Page does not display
The followings are the steps to determine the Portal Run-time Failure:
  • Review the SystemOut.log and SystemErr.log at location (/IBM/WebSphere/ /wp_profile/logs/WebSphere_Portal/)
  • Analyze these two files to see if there is any failure and try to find out what the causes of this failure are.
  • If the above problem determination approach did not resolve the issue, use the more advanced troubleshooting method by enabling additional tracing (Click here to see how to enable tracing).

Portal Start Time Failure


The following diagnostics logs and files are essential to troubleshooting an issue:

SystemOut.log. This file tells what activities happened during the Portal Starting time.
SystemErr.log. This file tells what exceptions/errors happened during the Portal Starting time.

The following are the steps to determine this issue:

  • Find if there is any Portal back-end connectivity problem, for example: LDAP or Portal Database.
  • Find if there is any WebSphere resource not working properly, for example: JMS, EJB, or JDBC
  • Find if there is any WebSphere Portal Service cannot be started.

WebSphere Portal Recovery Strategy


This section covers the strategies that can be taken into consideration if the analyses as mentioned above do not produce any meaningful action.

  • PORTAL RE-INSTALLATION : This step is the last step if all the previous steps fail. This step is basically portal re-installation from scratch.
  • PORTLET RE-INSTALLATION : This step is the Portlet installation and is taken if the failure only occurs to a specified Portlet.
  • PAGE RECONFIGURED : This step is the Page reconfiguration and is taken if the failure only occurs to a specified Page.
  • PORTAL LATEST WORKING BACKUP RESTORATION : This step is the Portal Restoration from the latest working backup. This is a major activity and will rollback all Portal changes that have been done to the Portal.

What is the best way to deploy portlets within Portal environment?

This article lists down different ways of deploying portlets within Portal environment listing advantages and disadvantages of each.

Deployment
Portlet Deployment


One Portlet per war

In this approach, each portlet is bundled as one war file.

Advantages:
  • Abstraction to the lowest level
  • Independent development and testing of portlets
  • Change to single portlet will not impact other portlets within portal
Disadvantages:
  • Messy deployment as the number of portlets grow
  • Sharing of portlet session is an issue
All portlets in a single War file

In this approach, we pack all the portlets in one war file.

Advantages:
  • Clean deployment, easy to manage
  • Easy to share portlet session
Disadvantages:
  • Parallel development is a challenge
  • The size of the deploying war becomes extremely large and takes a long time to deploy
  • Single change in one portlet will cause a redeploy of all portlets, adding re-testing overheads and making portal unavailable
Grouping portlets in one war

This approach refer to portlets grouped together based on function/ same functionality into single WAR files. 

Advantages:
  • Small changes to a functionality will not affect the overall portal
  • Clean and easy to manage
  • Portlets which need to share session will be grouped into one
  • Parallel development at functionality level is possible

Disadvantages: 
  • Maintenance at individual Portlet level might affect other packaged portlet of the application

Shared library management in ext of Portal server


Shared library in ext of Portlets

Shared Libraries

Apart from portlet applications, there are certain shared libraries used by most of the portlet applications (WAR's). This lead to challenge of how to deploy these shared applications. Here are few options:

Single Portlet application and shared libraries packed into one EAR 
This approach suggests having single portlet application, along with shared libraries and resource files, packed as one EAR file.

Advantages:
  • Each EAR will be truly independent
  • Common functionality could be extended based on the need of the portlet application
  • Changes to shared libraries will not impact all the other independent EAR's
Disadvantages: 
  • Each EAR would be large and will take longer deployment time
  • Each individual EAR would contain copy of the framework classes and maintainability of different versions of framework classes in different columns will be a challenge
  • More complex deployment process
  • Changes to shared libraries will require redeployment of all the EARs
  • Complex deployment
  • Class loading strategy of the application server would have to be restricted to PARENT LAST

Multiple Portlet applications and shared libraries packed into one EAR file 

This approach suggests that multiple portlet applications along with shared libraries and resources files, such as properties, packed into one single EAR.

Advantages:

  • All portlet applications share same common libraries
Disadvantages:
  • Big size and will take a long time for deployment
  • Change to single portlet application will require redeployment of whole EAR and thus unavailability of the application
  • Complex deployment

Shared library in ext of Portal server

This approach suggests that the shared library to be placed within the ext library directory of the portal server. This approach is suggested when there is are framework libraries which need to be shared across the organization (Organization level frameworks for logging/exception handling etc.) and each application hosted in the environment should follow those standards.

Advantages:
  • One of the most common way of sharing the files among the multiple applications
  • In portal world, all the common files pertaining to theme, portlets and third party application are stored in shared library
  • No need to use Dyna Cache or JSR 286 events to  save and share values across portlets
  • Library would be available across applications
  • Individual applications would be lighter since they won’t contain the code for framework classes
Disadvantages:
  • Change management for the shared libraries would be very risky
Steps for Creating a Shared library on Portal Server
  • Logon to Websphere Application server console
  • Navigate to Environment -> Shared Libraries
  • Click apply and save it
Assigning to a server 
  • Go to servers -> Application servers -> websphere portal -> Java and Process management -> classloader.
  • Click Shared library references
  • Add the library that you want to share
  • Save the configuration and restart the Portal server.
Note: Restart of server is mandatory for the class loader to load all the newly added class files. Now, you can add the shared jars to build path for proper compilation and deployment.