GBIF Integrated Publishing Toolkit (IPT) installation

http://code.google.com/p/gbif-providertoolkit/

 

JAVA and Tomcat

You will need Java and an application server like for example the Apache Tomcat installed.

• Java,  http://developers.sun.com/downloads/
• Apache Tomcat,  http://tomcat.apache.org/download-60.cgi

prompt$ java -version
prompt$ $CATALINA_HOME/bin/version.sh

Java environment settings

You should have JAVA_HOME and CATALINA_HOME set for your environment. [URL]

export JAVA_HOME=/YOUR_JAVA_LIBRARY_PATH_HERE/Java/Home (eg: /Library/Java/Home)
export CATALINA_HOME=/YOUR_TOMCAT_DIRECTORY_PATH_HERE/tomcat/Home (eg: /usr/local/tomcat/Home)

The default memory assigned to Java is not enough to run Tomcat. My example below assign 0.5 GB as start up memory, max 1 GB as maximum memory and also an increased Maximum permanent memory. 128 MB permanent memory works fine for me and increase the stability of the Tomcat significantly. [URL]

export JAVA_OPTS="-Xms512M -Xmx1024M -XX:MaxPermSize=128M"

----

Further comments

The default memory allocated to Java and thus to Tomcat is very low and need to be adjusted upwards for the IPT (or any Java application through Tomcat) to work properly.

To adjust the memory for Java to a higher level use the environment variable JAVA_OPTS. The initial java heap size (xms) and the maximum java heap size (xmx) can be set by the following setting:

export JAVA_OPTS="-Xms512M -Xmx1024M"

This is my setting for my laptop. For the server at NordGen we allocate more memory. The memory need to be an exact number of bytes. I suggest to Google for more information. Or read the IPT documentation.

But still after adjusting the heap memory up, the IPT was still unstable. Jose suggested to set MaxPermSize in addition to the heap size. This worked very well on the server. On the laptop I find this really consumes a lot of memory from other things and I rather prefer to restart the Tomcat when it gets stuck instead... ;-)

export JAVA_OPTS="-Xms512M -Xmx1024M -XX:MaxPermSize=512M"

CATALINA_OPTS will also work fine - assigning more memory to Tomcat (rather than in general to Java).

export CATALINA_OPTS="-Xms512M -Xmx1024M -XX:MaxPermSize=512M"

Both PermSize and MaxPermSize could be useful... Anyway the MaxPermSize setting made wonders for the stability of the IPT at NordGen.

export CATALINA_OPTS="-Xms512M -Xmx1024M -XX:PermSize=512M -XX:MaxPermSize=1024M"

On the laptop (MacBook) I am setting this in my .bash_profile - as I log in as myself and the system is running as me. On the server we read these variables into the environment. Typing the above command on a running Linux server would work fine, but need to be typed again after restart - so better to add to a script during startup. One option is of course simply to add this line to the Tomcat startup script...

If you have more memory, all the numbers above could be adjusted upwards. The example above is more the minimum I found to work ok on the laptop.

IPT Installation

• Download the ipt toolkit and the geoserver library from the IPT project home [URL] [URL].
• Rename the IPT war file as ipt.war and drop in the webapps folder of tomcat (or use the tomcat admin interface to deploy the ipt.war).
• Same with the geoserver library if not already installed for your system.
• Navigate with your web browser to http://localhost:8080/ipt/
• Login with username admin and password admin

GBIF IPT Customization

Customization of the IPT user interface.
For the GBIF GB16 NODES Training Workshop, 2009-10-04

The GBIF IPT (Integrated Publishing Toolkit) can be customized to your needs.
You may want to modify the CSS Style to follow the layout specifications of your organization.
You may also want to add a translation of the user interface for your own language.
Below are a few hints.


Web Icon (favicon.ico)

Log in to your IPT instance
Navigate in the menu to: Admin -> Configuration
  http://localhost:8080/ipt/admin/config.html

From this page you will find an input box named: "Additional raw HTML headers".
In this box add your icon, for example like the following:

<link rel='icon' type='image/vnd.microsoft.icon' href='http://localhost/images/icons/nordgen.ico' />

  or for example

<link rel='icon' type='image/png' href='http://localhost:8080/ipt/images/icons/nordgen.png' />

  and (for Internet Explorer)

<link rel="SHORTCUT ICON" href='http://localhost:8080/ipt/images/icons/nordgen.png'/>

Remember to add your icon image file at the URL you specify here...!

STYLE (CSS)

The GBIF IPT layout style definitions are made using Cascading Style Sheet.
The "theme.css" is starting the show:

 http://localhost:8080/ipt/styles/gbifn/theme.css

 http://localhost:8080/ipt/styles/gbifn/typo.css

 http://localhost:8080/ipt/styles/gbifn/layout.css

 http://localhost:8080/ipt/styles/gbifn/tables.css

 http://localhost:8080/ipt/styles/displaytag.css

 http://localhost:8080/ipt/styles/forms.css

 http://localhost:8080/ipt/styles/gbifn/forms_g.css

 http://localhost:8080/ipt/styles/messages.css

 http://localhost:8080/ipt/styles/gbifn/providertool.css

 http://localhost:8080/ipt/styles/gbifn/providertool2.css

Start for example with editing the file:

 webapps/ipt/styles/gbifn/layout.css

Or make yourself a new theme:

Copy folder:

  webapps/ipt/styles/gbifn/

to

  webapps/ipt/styles/nordgen/

and edit:

  webapps/ipt/styles/nordgen/layout.css

Next, edit:

  webapps/ipt/WEB-INF/web.xml

Around line 20 modify:

  <!-- Define the default CSS Theme -->

  <context-param>

   <param-name>csstheme</param-name>

   <param-value>gbifn</param-value>

  </context-param>

to:

  <!-- Define the default CSS Theme -->

  <context-param>

   <param-name>csstheme</param-name>

   <param-value>nordgen</param-value>

  </context-param>

LANGUAGE, translation of user interface


1. Add a flag for your new language:

  webapps/ipt/images/flags/flag_no.png

  webapps/ipt/images/flags/flag_se.png

2. Activate language select menu in:

  webapps/ipt/common/header.jsp

Around line 25, add (inside switchLocale div, ul block):

  <li><a href="?locale=no"><img src="<c:url value='/images/flags/flag_no.png'/>"/></a></li>

  <li><a href="?locale=se"><img src="<c:url value='/images/flags/flag_se.png'/>"/></a></li>

3. Add translated menu terms for your language:
Menu labels are defined in:

  webapps/ipt/WEB-INF/classes/ApplicationResources_no.properties

  webapps/ipt/WEB-INF/classes/ApplicationResources_se.properties

PostgreSQL shared memory settings

FATAL: connection limit exceeded for non-superusers

The shared memory settings are mostly set too low for the PostgreSQL database server
on most systems. In particular on systems with BSD heritage.
When PostgreSQL exceeds one of the various hard shared memory and
semaphores (IPC) limits, the server will refuse to start.

The most important shared memory parameter is SHMMAX, the maximum size, in
bytes, of a shared memory segment. If you get an error message from
shmget like Invalid argument, it is likely that this limit has been exceeded.

You can see the current shared memory values with:
$ sysctl kern.sysv.shmmax
$ sysctl -a | grep shm

You can play around with the values temporarily (until restart) on the command line
$ sudo sysctl -w kern.sysv.shmmax=536870912
$ sudo sysctl -w kern.sysv.shmall=131072


Configuration parameters affecting PostgreSQL's shared memory usage:

• max_connections
• max_prepared_transactions
• shared_buffers
• wal_buffers
• max_fsm_relations
• max_fsm_pages

While it is possible to get PostgreSQL to run with SHMMAX as small as
1 MB, you need at least 4 MB for acceptable performance, and desirable
settings are in the tens of megabytes.

Lowering the number of allowed connections is a temporary workaround for failures.

---

In OS X 10.3.9 and later, you may create a file named ‘/etc/sysctl.conf’,
containing variable assignments such as:

kern.sysv.shmmax=536870912
kern.sysv.shmall=131072
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8


My original settings (before modifications) was:
kern.sysv.shmmax=4194304
kern.sysv.shmall=1024
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8

This method is better than editing ‘/etc/rc’ because your changes will be preserved across system updates.
Note that all five shared-memory parameters must be set in ‘/etc/sysctl.conf’, else the values will be ignored.
Beware that recent releases of OS X ignore attempts to set
SHMMAX to a value that isn't an exact multiple of 4096.
SHMALL is measured in 4 kB pages on this platform.

To check your settings you may issue the command:
$ cat /etc/sysctl.conf

In all OS X versions, you'll need to reboot to make changes in the
shared memory parameters take effect.

---
In OS X 10.3 and later, the commands is set in ‘/etc/rc’ and must be edited there. Note that
‘/etc/rc’ is usually overwritten by OS X updates so you should expect to have to redo your editing
after each update.
---
In OS X 10.2 and earlier, edit the file
‘/System/Library/StartupItems/SystemTuning/SystemTuning’
and change the values in the following commands:
"sysctl -w kern.sysv.shmmax=..." (shmmax, shmin, shmmni, shmseg, shmall)
---

Linux

The default max segment size is 32 MB. You may want to increase this value for larger PostgreSQL installations. The remaining defaults are quite generously sized, and usually do not require changes. The max segment size can be changed via the sysctl interface. For example, to allow 128 MB, and explicitly set the maximum total shared memory size to 2097152 pages (the default):

$ sysctl -w kernel.shmmax=134217728
$ sysctl -w kernel.shmall=2097152

In addition these settings can be saved between reboots in ‘/etc/sysctl.conf’.

---
http://www.google.se/search?q=mac+os+x+shmmax
http://www.caktusgroup.com/blog/2009/08/13/setting-postgresqls-shmmax-in-mac-os-x-105-leopard/
http://www.network-theory.co.uk/docs/postgresql/vol3/SharedMemoryandSemaphores.html
http://raam.splitbook.com/?p=13

Adding an extension for germplasm, GBIF IPT

The GBIF Provider Toolkit (IPT) is a database wrapper to help the provider of biodiversity information to set up and provide these data as a web service. The primary data exchange standard deployed by the GBIF IPT is the Darwin Core standard maintained by the Biodiversity Information Standards (TDWG).

http://code.google.com/p/gbif-providertoolkit/
http://code.google.com/p/darwincore/

Below are the steps I followed to develop a draft germplasm extension to the Darwin Core (DwC) standard, and next to add this draft DwC extension to the GBIF IPT software. You may follow the same steps to develop your own extension and to add this to the GBIF IPT.

If you are interested, you may visit the IPT from NordGen with a DEMO dataset mapped to terms from this DRAFT Darwin Core extension for germplasm at:
http://ipt.nordgen.org/ipt/occResource.html?resource_id=77

Summary of steps:

1. Schema
I built a suggestion to a DRAFT Darwin Core extension for germplasm
http://code.google.com/p/darwincore/source/browse/trunk/xsd/profiles/germplasm/dwc_germplasm_0_1.xsd
http://rs.nordgen.org/dwc/germplasm/0.1/dwc_germplasm_0_1.xsd

2. XML definition for IPT
Listing the terms to use from the suggested DRAFT DarwinCore extension
http://rs.nordgen.org/dwc/germplasm/0.1/ipt_germplasm_0_1.xml

3. Add this XML definition to a copy of extensions.json
Starting with: http://gbrds.gbif.org/registry/ipt/extensions.json
Add text {"title":"Germplasm","url":"http://rs.nordgen.org/dwc/germplasm/0.1/ipt_germplasm_0_1.xml","id":9}
Saved as: http://rs.nordgen.org/rs/ipt/extensions.json

4. Modify ipt.properties
PATH_TO_TOMCAT_WEBAPPS/ipt/WEB-INF/classes/ipt.properties
Line 7: "registryUrl=http://gbrds.gbif.org/registry" modified to
"registryUrl=http://rs.nordgen.org/rs"

5. Restart Tomcat (reload ipt)
The new ipt.properties settings does not seem to be loaded without restart of tomcat

6. Add/load the new extension to IPT
Login with admin permissions
http://your-ipt-service-url/ipt/admin/extensions.html
Click "Check for all available extensions"
Select the name of the new extension
Select "install" and "Done"

7. Mapping of terms to your data
Select the new extension under "Add New Mapping"
Click "Add" button to the right
Select and map terms

Remarks:

The IPT will contact the registry as defined in "ipt.properties". By default the GBIF registry in Copenhagen is defined here, but you may modify this to point to your own registry. I did not mimic the complete features of the registry - ONLY the list of extensions "extensions.json" to add the new draft germplasm extension I made. You may want to add your own thesaurus "thesauri.json" of controlled vocabulary or more features in the similar way. After loading my new extension to my IPT at NordGen, I modified the "properties.ipt" back to point to the GBIF registry in Copenhagen again.

To mimic the registry feature for adding the extension, I made a copy of the file from http://gbrds.gbif.org/registry/ipt/extensions.json and added my application schema from the extension as you will see at http://rs.nordgen.org/rs/ipt/extensions.json.

The application schema (as defined in "extensions.json") and online from: http://rs.nordgen.org/dwc/germplasm/0.1/ipt_germplasm_0_1.xml, creates a namespace (for the use by IPT) as well as includes the URL for the actual draft germplasm extension I made:

http://rs.nordgen.org/dwc/germplasm/0.1/ http://rs.nordgen.org/rs/dwc/germplasm/0.1/dwc_germplasm_0_1.xsd

The application schema itself (ipt_germplasm.xml) only lists the terms I want to use in IPT, with the property name to be displayed in the IPT mapping interface. To create this file I started with one of the other extensions http://gbrds.gbif.org/resources/extensions/vernacularName.xml. I just made a copy of this file and replaced the names and terms for vernacular name extension with the terms I wanted from the draft germplasm extension.

NB! This DRAFT extension for germplasm is still under development and I am sure that I still duplicate some terms from DarwinCore or perhaps other extensions! I did not yet work much with the terms for trait observations on germplasm, but added most of the terms from the new proposed draft standard for these data for the European genebank community.

Open dBase file from an ESRI shape on Mac Os X

As part of the ESRI shape file format, there is a dBase (dbf) file with data in spreadsheet format. MS Excel for Mac (2004, 2008) will not open this file (MS Excel for Windows open dBase files, the Mac version seems not to do so).

I open the dBase dbf file from the ESRI shape (from DIVA GIS) with OpenOffice.
org 3.0.0 on a MacBook with OS X Leopard Version 10.5.6. This works fine for
me and I can save the file for example to an MS Excel format to open with MS
Excel 2008 for Mac (or directly to the CSV format). The dBase file does not
open well with MS Excel 2008 even with the "All Documents" trick. Left click
open with OpenOffice works fine.

You may also want to try the command line application: dbf-0.8.3.1, dbf-0.9.0-Alpha-1

OS X system upgrades and my Apache HTTPD + PHP

OS X Leopard and compiling Apache HTTPD + PHP

I have a MacBook with my own build for the Apache HTTPD + PHP. I need features for my PHP system not included in the generic bundle with OS X Leopard (PostgreSQL). Some of the add-ons for PHP are 32 bit, so I need to build my own 32 bit Apache HTTPD. For convinience I wanted to compile the new version of Apache HTTPD and PHP to the folder directory as the default bundled OS X versions. This provides easy management from the System Preference panel etc.

Apache HTTPD
Download the latest source code from: http://httpd.apache.org/download.cgi.
Use the command line prompt with a terminal window.

Uncompress for example to /usr/local/src/

• tar xzvf http.2.2.11.tar.gz (option z for gz compression file.tar.gz)
• tar xjvf http.2.2.11.tar.gz (option j for bzip2 compression file.tar.bz2)

Enter the source code directory: (cd /usr/local/src/http.2.2.11/)

Give your local configure option for the compilation. I use:
./configure \
--enable-layout=Darwin \
--enable-mods-shared=all \
| tee INFO_httpd_configuration.txt

The option --enable-layout=Darwin provides the trick to compile your own version of the HTTPD web server to the very same directory location as the OS X Leopard web server (/usr/local/apache2/, /usr/libexec/apache2/). You may want to create a backup of these directories!

The pipe (|) to tee is NOT neccessary, but give a handy text file with a log of the output to standard output during the execution of the configuration command. The tee application will let the standard output through to the screen as well as also to the text file you specify.

Next make and make install:
make | tee INFO_httpd_make.txt
sudo make install | tee INFO_httpd_make_install.txt


PHP
Compiling PHP exactly as you will require some efforts to get all the external libraries and support applications you want to have installed. Fink and MacPort may be very useful if you have problems to compile them yourself, or if you simply do not require any particular configuration settings.

My configuration settings:

'./configure' \
'--enable-layout=Darwin' \
'--with-config-file-path=/etc' \
'--sysconfdir=/private/etc' \
'--disable-dependency-tracking' \
'--enable-dbase' \
'--enable-cli' \
'--enable-exif' \
'--enable-ftp' \
'--enable-mbregex' \
'--enable-mods-shared=all' \
'--enable-mbstring' \
'--enable-soap' \
'--enable-sockets' \
'--enable-trans-sid' \
'--infodir=/usr/share/info' \
'--localstatedir=/var' \
'--mandir=/usr/share/man' \
'--prefix=/usr' \
'--with-apxs2=/usr/sbin/apxs' \
'--with-curl=/usr' \
'--with-gd' \
'--with-iodbc=/usr' \
'--with-jpeg-dir=/usr/local/bin' \
'--with-kerberos=/usr' \
'--with-ldap=/usr' \
'--with-mysql=/usr/local/mysql' \
'--with-mysql-sock=/var/mysql' \
'--with-mysqli=/usr/local/mysql/bin/mysql_config' \
'--with-pdo-mysql=/usr/local/mysql' \
'--with-pdo-pgsql=/usr/local/pgsql/bin' \
'--with-pgsql=/usr/local/pgsql' \
'--with-png-dir=/usr/local/bin' \
'--with-tidy=/opt/local' \
'--with-xmlrpc' \
'--with-xsl=/usr' \
'--with-zlib' \
'--with-zlib-dir=/usr' \
'--without-pear' \
| tee INFO_php_configuration_log.txt

Next make and make install:
make | tee INFO_php_make.txt
sudo make install | tee INFO_php_make_install.txt

With --enable-layout=Darwin the compiled PHP libraries will find their way to the save directory location as the default bundeled PHP with OS X Leopard. If you prefer to be more careful, you may compile your PHP without the "--enable-layout=Darwin" and instead for example "--prefix=/usr/local/php5". In the sub-folder "libs" in the source code folder were you compiled PHP you will find "libphp5.so". You may manually copy this to "/usr/libexec/apache2/" if the library file does not arrive there during the make install.


OS X system upgrades
Most of the OS X system upgrades destroys my own web server compilation. The central problem seems to be that a generic OS X HTTPD of 64 bit replaces my own HTTPD. The fix is however rather simple. Save and keep the source code you used to compile your HTTPD web server, for example in /usr/local/src/httpd.2.2.11/. Execute make install from the source code folder. If a new version of HTTPD is available, I usually download this one and run the ./configure && make && make install as described above.

You may check your version of the Apache HTTPD web server from the command prompt with: httpd -V. Look for the line "Server version: Apache /2.2.11" for your web server version and the "Architecture: 32-bit" or "Architecture: 64-bit". You may also want to verify the local of the httpd application with the command: which httpd. The default location for OS X Leopard is /usr/sbin/httpd. If you have other httpd application from for example fink or MacPorts you may see another path to the httpd. You may check your PATH with: echo $PATH.