The Nimrod Portal Manual

Introduction - Installation

Contents

Installation requirements	A description on what is required to run the portal.
Installation location	Information in where to install the portal.
Installation from source	Instruction on how to compile binaries from the source.
Install binaries	Instructions on how to run the portal.
Configure WWW install	Steps required in configuring the portal.
Configure remote WWW install	Steps required in configuring a remote web server, separate to the portal/Nimrod server.
Set up the Nimrod Portal database	The Nimrod Portal database
MOTD and the disclaimer	Information on the MOTD and the Disclaimer.
Configuring the Nimrod bank	How to configure Nimrod to use the Nimrod bank and configuring resources to report their cost.
Configuring Nimrod to use user certificate authentication	Enable users to validate with certificates instead of using their username and password.
Diagnosing problems	What to check and turning on debugging modes.
Common problems	Some common problems of the installation.

Installation location

top

The Nimrod Portal does not dictate the installation location. We suggest you install it into /local/nimrodportal.

The Portal does not have an installation program so we suggest you unpack the distribution directly into the install directory.

As the users will never need to explore this directory, we suggest you chmod 711 nimrodportal to discourage curious users.

Installation requirements

top

The Nimrod Portal has three mandatory requirements. A web server, PostGres SQL and Nimrod.

The Nimrod Portal has been tested under Apache web server on Linux, SunOS and NetBSD with out any changes to the source. As the web server is only used to launch the portal's CGI script, there should not be many problems on other systems.

These instructions assume that the web server is installed on the Nimrod server. This is the easiest way to install the Nimrod Portal, however, it is possible to have the web server separate from the Nimrod install. We suggest you follow the install instructions and install on the Nimrod server. Then examine the two example files, np.ssh.cgi.example and np.sshd.example in the sbin directory. You will need a ssh key pair without a passphrase to connect to a user without rights on the Nimrod server.
NOTE: We have observed that ssh can use a lot of load on the web server. Please ensure that you have a device called /dev/*random and that ssh is enabled to use it.

As Nimrod's database backend currently is PostGres SQL, the Portal has been developed with PostGres. However, it may work with other SQL implementations, but have not found the need as Postgres is already installed with Nimrod.

The portal assumes that postgres has been configured with a default database set for every user. So when the Nimrod Portal calls upon Nimrod, Nimrod will connect to the user's default database. Postgres normally does not have a problem as it normally uses the user's name as the default database name. But we have had some installations that have no default.

Although the portal requires Nimrod, it does not need Nimrod to install. If the portal has been correctly installed, it will constantly report errors to the user until Nimrod is installed on the server.

Installation from source

top

Installation from the source requires very little customisation as it does not depend on many other system libraries. In most cases, it requires a make install (or gmake install), however for compatibility, there are a few configurable options in how the users password is checked. There are also some options on which type of resources the user is allowed to add.

Instructions:

• Comment out PASSWD_SHADOW in the root directories Makefile if your system is not using shadow passwords or the OS automatically redirects passwd requests to the shadow password system (E.g. NetBSD). If unsure, do not comment out this line as "make" will not work this way if it is incorrect.
• Comment out PASSWD_SHADOW_H in the root directories Makefile if your system does not require the shadow.h file included. This will be the case if you have commented out the PASSWD_SHADOW option. If unsure, do not comment out this line as "make" will not work this way if it is incorrect.
• Comment out PASSWD_CRYPT_H in the root directories Makefile if your system does not require the crypt.h file included. If unsure, do not comment out this line as "make" will not work this way if it is incorrect.
• Comment all the above items and uncomment PAM_LIBRARY and USE_PAM in the root directories Makefile if you require PAM authentication. This is required for MAX OSX.0.3. You will also need to make a nimrodportal file in the pam.d directory. You may use the login file in pam.d as an example.
• Other options may need to be set if they are not in the path (E.g., JAVAC, JAR).
• As the root user, run make ; make install (or gmake ; gmake install). Note: The install directory for make install is the source directory. Also note: Under some Linuxes sudo make install will not have the permissions to set the SUID bit on the nimrodportal.cgi. Please make as the root user.

The directory where you have installed the portal files does not need to be added to the system path. However, if you have following external packages that integrate with the portal, you will need to copy them to the ${NIMRODPORTAL_INSTALL}/bin directory.

These packages are:

Nimrod/O	If you have Nimrod/O version 2.5 or greater, copy the nimrodo and the Nimcache (and gamut if setting up Nimrod/OI) executables to the nimrodportal's bin directory. Instruction on configuring Nimrod/O for the portal are located in the Set up the Nimrod Portal Database section.
EnFuzion	The Nimrod Portal can use the EnFuzion enfgenerator as an alternative to the Nimrod's one. This version gives better feedback when there are errors in the user's .pln files. Copy enfgenerator to the nimrodportal's bin directory. No further configuration is required.

Now that you have the portal configured, you'll need to set up the cgi script and Java/Image directories.

Configure Web install

top

This section will cover the steps to setting the portal up on your https server. The Portal has no restrictions on the URLs used to access the CGI script nor on the images/npapplets directories. In fact they can be located on different servers.

It is strongly advised that the portal is installed on an https server. When the user logs in, a clear text password is sent to the portal server. Using https, this is encrypted.

Setting up the images and npapplets

A note about Nimrod Databases

Configuring the CGI script

The options descriptions are:

NIMROD_INSTALL	This must point to the location of where Nimrod is installed.
PSQL_INSTALL	This must point to the location of where postgres in installed. It must be one directory down of its bin directory. E.g. if the psql command is in /usr/bin/psql, then this option must be /usr
NIMRODPORTAL_INSTALL	This must point to the location of where the nimrod portal has been installed. It must be one directory down from the bin directory.
NIMRODPORTAL_URL	This must be the URL to parent of the npapplet and the images directory.
NIMRODPORTAL_BIN_URL	This must be the URL to this CGI script. The Nimrod Portal uses this to generate the URLs.
GLOBUS_LOCATION	This must be set to the path of the Globus install. Both Nimrod and the Portal use this path to use Globus resources.
PBS_LOCATION	This must be set to the path of the PBS bin install. Nimrod uses this path to work with PBS.
CONDOR_CONFIG	This must be set to the full path of the Condor config file. Nimrod uses this file to work with Condor.
SGE_ARCH and SGE_ROOT	Nimrod uses these variables to work with SGE.

There are other options that may need to be changed in non-standard installations.

#!/usr/bin/env bash ##################################################################### # # This is the example file for the main cgi script used for the # Nimrod Portal. It needs to be customised to your resource. # ##################################################################### # Options used by Nimrod # Database type NIMROD_DATABASE=pgsql # Nimrod install NIMROD_INSTALL=/opt/nimrod-3.2.0 # The PSQL installation directory PSQL_INSTALL=/usr PSQL_LOCATION=${PSQL_INSTALL}/bin # The bin directory of qsub for PBS PBS_LOCATION=/usr/pbs/bin # The Condor config file CONDOR_CONFIG=/usr/condor/etc/condor_config # The type of sun grid engine installed SGE_ARCH=glinux # The root directory SGE SGE_ROOT=/opt/gridengine # Options used by the Portal # This install directory of the portal. NIMRODPORTAL_INSTALL=/opt/nimrodportal # This is the common URL for the images and applets. The portal will # append /images/ or /nvapplet/ to this URL. NIMRODPORTAL_URL=https://nimrod.csse.monash.edu.au # This is the URL for this script. NIMRODPORTAL_BIN_URL=https://nimrod.csse.monash.edu.au/cgi-bin/np.cgi # This is the command the Portal will execute to establish a connection # to the SQL database. This is done as root. The results back must # be comma separated. The portal also expects that newline will # terminate each request. The following example should be on one line. # And the comma must appear exactly after the -F NIMRODPORTAL_SQL_CMD="${PSQL_LOCATION}/psql -t -A -F, -U nimrodportal -d nimrodportal -S" # Globus install path GLOBUS_LOCATION=/opt/globus/2.4 #################################################################### # # Options below may only need to be modified if you are having # problems with library paths etc. # #################################################################### export NIMROD_DATABASE export NIMROD_INSTALL export NIMRODPORTAL_INSTALL export NIMRODPORTAL_URL export NIMRODPORTAL_BIN_URL export NIMRODPORTAL_SQL_CMD export PSQL_INSTALL export PSQL_LOCATION export GLOBUS_LOCATION export PBS_LOCATION export CONDOR_CONFIG export SGE_ARCH export SGE_ROOT PATH=$NIMROD_INSTALL/bin:$PATH:/bin:/sbin:/usr/bin:/usr/sbin PATH=$PATH:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin PATH=$PATH:$GLOBUS_LOCATION/bin:$PBS_LOCATION export PATH PATH=$PSQL_INSTALL/bin:$NIMRODPORTAL_INSTALL/bin:$PATH ; export PATH LD_LIBRARY_PATH=$GLOBUS_LOCATION/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH LIBPATH=$LIBPATH:$LD_LIBRARY_PATH ; export LIBPATH # Check that directory structure is accessible if [ ! -x $NIMRODPORTAL_INSTALL/bin/nimrodportal.cgi ] then echo Content-type: text/plain echo echo -n User \'`whoami`\' with group access\(es\) \'`groups`\' echo is unable to execute \'$NIMRODPORTAL_INSTALL/bin/nimrodportal.cgi\' exit fi # Launch the portal. exec $NIMRODPORTAL_INSTALL/bin/nimrodportal.cgi 2>/dev/null

Configure remote Web install

top

Nimrod and the Nimrod Portal can be installed on a server separate to the web server. These instructions should only be followed if you have Nimrod installed on a separate machine to your web server. This can be done by using SSH to tunnel the WWW connection to the Nimrod Portal installed on the Nimrod Server. This method can also be used using one machine to avoid problems with SELINUX.

The location of installed programs looks like this:

Computer A	Computer B
Web server (e.g. Apache)	Nimrod/G Nimrod Portal

Create a Nimrod Portal user on Computer B

Setting up the scripts and SSH files

Computer A	Computer B
All files on Computer A are located in the cgi-bin directory and must be owned by the web-server user and readable by only the owner (chmod og-rwx known_hosts nimrodportal_id): np.ssh.cgi known_hosts (contains the host key for Computer B) nimrodportal_id (SSH user private key)	${NIMRODPORTAL_INSTALL}/bin/np.sshd ${HOME}/.ssh/authorized_keys2 (contains the public key)

Obtaining these files can be done this way:

• known_hosts: This file can be created by sshing into computer B on any account. After you successfully connect, your ${HOME}/.ssh/known_hosts file will contain the public key of Computer B. You can then copy the known_hosts file from the account you sshed with into the cgi-bin directory that contains np.ssh.cgi.
• nimrodportal_id and ${HOME}/.ssh/authorized_keys2: These two files will have to be generated. We suggest you do this under the new user account created on Computer B. As the user, run this command ssh-keygen -tdsa and when asked for a passphrase, you must leave blank. This will create two files, id_dsa and id_dsa.pub. id_dsa must be copied to Computer A's cgi-bin directory as nimrodportal_id (where the np.ssh.cgi script is) and must be protected with chmod og-rwx nimrodportal_id. id_dsa.pub must be copied to ${HOME}/.ssh/authorized_keys2 as the user on Computer B.

  To test this, on Computer A change into the directory where the np.ssh.cgi script is and type this command:
  ssh -o BatchMode=yes -o UserKnownHostsFile=known_hosts -x -i nimrodportal_id <nimrod portal username>@<nimrod portal host> echo test
  The word "test" should be returned. If not, please make sure you have the correct known_hosts and nimrodportal_id files.

Configuring the Web server (Computer A)

You will also need to set some environment settings in the np.ssh.cgi to instruct the script where the Nimrod Portal is installed. The two setting are: SSH_DESTINATION (in the form of username@nimrod.server) and NIMRODPORTAL_INSTALL which is the location of where the Nimrod Portal is installed on the Nimrod server (Computer B).

Configuring the Nimrod Portal scripts (on Computer B)

If successful, all connections to the cgi script on the web server (Computer A) will then redirect down SSH to Computer B's Nimrod Portal install.

Set up the Nimrod Portal database

top

The Nimrod Portal uses a Postgres database to hold miscellaneous information used at run time (such as user logged in information). It also holds configuration items such as Administrators and plugins. This section will cover configuring the Nimrod Portal to add Administrators and to add plugins. By default there are no plugins (including Nimrod/G).

Creating the Nimrod Portal user and database

createuser -A -D nimrodportal

Creating the database is described below. The nimrodportal user and database must be protected from other users. Here are a few tips:

• Please read up about PostgresSQL user accounts from their website www.postgresql.org.
• Create a Postgres user called nimrodportal and protect that user from other users. Make sure another user cannot become the nimrodportal database user by doing a psql -U nimrodportal command. Tips on how to protect this user in apache are in the next section. For more advanced users, you can change the way the Nimrod Portal accesses the database by change the NIMRODPORTAL_SQL_CMD variable in the np.cgi file.
• Create a database called nimrodportal and protect it from other users. Users with privileges can connect to the database with the command psql -d nimrodportal, but make sure other non-privileged users can't connect.
• See section below for tips on how to protect the nimrodportal user and database from other users.

Configuring postgres configuration files

This line must appear in the pg_hba.conf file before any other match. That is, it must be before the "local any any" line.

pg_hba.conf

local nimrodportal nimrodportal ident nimrod

As the Nimrod portal is launched via the apache user and then the super user ID is set, you may need to map both apache and the root user to map into the nimrodportal database user.

pg_ident.conf

nimrod root nimrodportal nimrod apache nimrodportal

Initialising the Database

• The database creation, done by the script npcreatedb in the sbin directory. This script may error as it tries to create the database you may of already created. You can ignore this error.
• Adding administrators. This is done by running the addadmin script in the sbin directory with the username as the first and only parameter. Adding a user as an administrator gives the user the power to become any other portal user. Any number of users can be an administrator, include none.
• Adding the plugins. There are three different types of plugins included with the Nimrod Portal distribution. They are:
  • Experiment types: At the very least, one experiment type plugin has to be installed. Failing this the Portal cannot do anything with experiments. The three current experiment plugin scripts are nimrod_plugin, nimrodO_plugin (Requires Nimrod/O 2.5) and nimrodOI_plugin (Requires Nimrod/O 2.5 with the Nimrod/O 'gamut' extension).
  • Resource types: At the very least, one resource type plugin has to be installed. Failing this, the portal cannot create new resources. The current resource types that the portal supports are fork_plugin, pbs_plugin, sge_plugin, condor_plugin, globus_plguin and the gt4_plugin.
  • Extensions: The other plugins provided are the GridWorld_plugin which requires the GridWorld applet in the npapplet directory, Java_plugin a link to the Java website, the usercert_plugin to enable to user to download a web browser certificate and the manual_plugin link.

All of these scripts read the sbinenv.conf file to find out the location of the Postgres commands. You may need to edit this file to include the full paths of the commands.

We suggest you create a script called npinit and have something like this in it:

#!/bin/sh # Init the database ./npcreatedb # Add the administrators ./addadmin cme ./addadmin slavisa # Add plugins ./nimrod_plugin ./nimrodO_plugin ./nimrodOI_plugin ./Manual_plugin ./GridWorld_plugin ./globus_plugin ./fork_plugin ./pbs_plugin

Congratulations: You are now ready to log into the Nimrod Portal!

MOTD and the Disclaimer message

top

Message of the day

The command resetmotd located in the sbin directory will reset all users to see the motd again. This should be run after the file is updated so that current loggedin users will see the new message.

Disclaimer message

The Nimrod Portal cannot be used by a user until the user accepts.

The command resetdisclaimer located in the sbin directory will reset all users to be prompted to acknowledge again.

Configuring the Nimrod bank services

top

Installing the plugin

This will cause the Nimrod Portal to try to contact the bank to determine if the user has enough credit. To configure which bank to use, you will now configure the command the Portal will use to contact the bank. This is done in the $NIMRODPORTAL_INSTALL/bin script called accessbank. The example installed will only need the hostname and path to the bank's scipt updated. (Note, you may remove the globus command and host and use your locale server as the bank server, however, user's still must have a default proxy in order to be identified by the bank)

You will also need to add yourself as a bank manager. To do this, add your Globus certificate subject to the file $NIMRODPORTAL_INSTALL/bankdata/manager. Your subject must be quoted. Once you have done this, you can modify other user's balances under your balance section in the resource management section on the Portal.

Managing user's bank balance.

Configure a remote resource to report its cost.

The Nimrod Portal will run the command located in the file /etc/nimrodresourcecost.conf on the remote resource. If this file or the command that it refers to is not there, then no cost information will be reported.

Example of the /etc/nimrodresourcecost.conf file:

/usr/bin/nimrodresourcecost

The output from the program /usr/bin/nimrodresourcecost is of expected format. For a resource that has no quota, an example script will look like:

#!/bin/sh echo CostPerHour: 20 echo UserQuotaHoursLeft: -2 echo QuotaExpDate: Never

The only no obvious setting is UserQuotaHoursLeft being -2. -2 represents unlimited quota. It is not used directly by Nimrod, but displayed to the user for the user to decide how to use the resource.

A more complicated example may look like this:

#!/usr/bin/env bash echo CostPerHour: 10 /opt/rash/bin/quotasu -P g14 | /bin/awk '{if ($1 == "Service" && $2 == "Units" && $3 == "remaining") print "UserQuotaHoursLeft: " $4}' /opt/rash/bin/nf_limits | /bin/awk 'BEGIN{doit=0}{ if (doit == 1 && $2 != "") print "QueueRatio-" $1 ": " $2}{if (doit == 1 && $1 == "---------------------------------------------------------------------------") doit = 2}{if (doit == 0 && $1 == "---------------------------------------------------------------------------") doit = 1}' MONTH=`/bin/date '+%m'` if [ $MONTH -le 3 ] then echo QuotaExpDate: 31-Mar-`date '+%Y'` else if [ $MONTH -le 6 ] then echo QuotaExpDate: 30-Jun-`date '+%Y'` else if [ $MONTH -le 9 ] then echo QuotaExpDate: 30-Sep-`date '+%Y'` else echo QuotaExpDate: 31-Dec-`date '+%Y'` fi fi fi

This example is taken from the National Facility's lc cluster at APAC. This first section of the script outputs the cost to use the resource. The next line runs a command to determine how many CPU hours the user has left in their quota. The third command outputs the cost ratio for each of the queues. (E.g. the express queue costs three times the amount of the default queue). The last section determines when the user's quota will expire. In the case of the lc cluster, every three months. This script will output this:

CostPerHour: 10 UserQuotaHoursLeft: 50.00 QueueRatio-express: 3 QueueRatio-normal: 1 QueueRatio-bonus: 0 QueueRatio-interactive: 2 QueueRatio-copyq: 1 QuotaExpDate: 31-Dec-2004

The Nimrod Portal expects these values in this format. To check if the script is working, you can always select the "Collect information" button on the "Resource Management" page.

Configuring Nimrod to use user certificate authentication

top

Come back later.

Diagnosing problems

top

Check the log

A few things to try

Portal Page	Nimrod command(s)
Experiments	nimrod portalapi getexperiments
An experiment	nimrod portalapi getexpinfo <expname> nimrod portalapi getdeadline <expname>
Creating an experiment (also used in reset)	nimrod portalapi addrun <expname> <exptype>
Deleting an experiment (also used in reset)	nimrod portalapi delrun <expname>
Saving a plan file	nimrod generate <plan file>
Starting an experiment	nimrod portalapi startexp <expname>
Stopping/Pausing an experiment	nimrod portalapi stopexp <expname>
Setting a deadline in an experiment	See Nimrod Manual about: nimrod portalapi setdeadline
Allocating a resource to an experiment	nimrod portalapi addserver <expname> <host> <resource type> nimrod portalapi removeserver <expname> <host> <resource type>
Adding/modify a resource	nimrod resource add <resource type> <host> nimrod portalapi setarch <resource type> <host> <arch> nimrod portalapi setgproxy <resource type> <host> <certificate>
Removing a resource	nimrod resource remove <resource type> <host>
Checking a resource	nimrod resource check <resource type> <host>
The Nimrod Job viewer applet	nimrod enfapi <Experiment name> <EnFuzion API call>

The Portal also maintains Globus credentials. Pleas ensure that these command work at the user's prompt.

Portal Page	Globus command(s)
Resource Management Page	$GLOBUS_LOCATION/etc/globus-user-env.sh grid-cert-info -subject -file <Certificate file name> grid-cert-info -enddate -file <Certificate file name> grid-proxy-info -subject -file <Proxy certificate file name> grid-proxy-info -timeleft -file <Proxy certificate file name>
Create Proxy	$GLOBUS_LOCATION/etc/globus-user-env.sh grid-proxy-init -cert <Certificate file name> -key <Certificate key's file name> -out <Destination proxy file name> -pwstdin -bits 1024 -hours <hours>

Turning on full logging

This log file has been set up for debugging purposes and does not have a friendly interface. Please follow these instructions carefully and the portal will crash without giving any reason why if this is done wrong.

Follow these instructions after installing the portal. The log file must be created and must be world readable and writeable, so we need to create the log file first. Do:

cd log
touch runtime.log
chmod 666 runtime.log
chmod a+x .

$NIMRODPORTAL_INSTALL/bin/nimrodportal.cgi 2>/dev/null

$NIMRODPORTAL_INSTALL/bin/nimrodportal.cgi 2>> $NIMRODPORTAL_INSTALL/log/runtime.log

echo -n "" > runtime.log

To turn off logging, change the line we modified in the np.cgi script back to what is was. We recommend this as the log file gets large very quickly.

Common problems

top

libglobus_gsi_proxy_core_gcc32.so.0 errors:

psql (pg_wrapper): No database specified:

User 'apache' is unable to execute '$NIMRODPORTAL_INSTALL/bin/nimrodportal.cgi':

top