FOSS4G Benchmark - Revision history

Wiki-Leo: Reverted edits by Domtheo (talk) to last revision by Arnulf Christl

2013-06-28T09:19:32Z

Reverted edits by Domtheo (talk) to last revision by Arnulf Christl

Wiki-Domtheo at 04:10, 28 June 2013

2013-06-28T04:10:07Z

Arnulf: /* FOSS4G Performance Shoot-out */

2013-06-04T13:23:08Z

FOSS4G Performance Shoot-out

Arnulf: /* FOSS4G Performance Shoot-out */

2013-06-04T13:21:59Z

FOSS4G Performance Shoot-out

Sourcepole: Update for Benchmark 2009

2010-08-23T07:19:11Z

Update for Benchmark 2009

Wiki-Camerons: typo

2010-04-05T22:45:01Z

typo

Sourcepole: SVN link

2010-03-08T13:57:44Z

SVN link

Sourcepole: FOSS4G WMS Benchmark page

2010-03-08T13:54:35Z

FOSS4G WMS Benchmark page

New page

== FOSS4G WMS Benchmark ==

== Introduction ==

The FOSS4G WMS Benchmark tests how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.

== FOSS4G Performance Shoot-out ==

Following a WMS server performance comparison between Mapserver and GeoServer at FOSS4G 2007 in Victoria, a performance shoot-out presentation at the yearly FOSS4G was institutionalized.

One of the most important 'rules of engagement' for each Web server team is that all parties must contribute any changes that they make to their software for this exercise, back to their community.

* [[Benchmarking_2010 | FOSS4G 2010 Barcelona]]
* [[Benchmarking_2009 | FOSS4G 2009 Sidney]]
* FOSS4G 2008 Cape Town
* [http://2007.foss4g.org/presentations/view.php?abstract_id=120 FOSS4G 2007 Victoria]

== Running the FOSS4G WMS Benchmark ==

The FOSS4G is a useful test suite for your own WMS server installation. This section gives you step-by-step instructions for running the FOSS4G WMS benchmark on different platforms.

=== Ubuntu 9.10 "Karmic Koala" ===

==== Install scripts and data sets ====

Needed packages:
sudo apt-get install subversion unzip
sudo apt-get install jmeter jmeter-http jmeter-junit
sudo ln -s /usr/share/jmeter/ /usr/share/jmeter/lib/junit

Get the test scripts and configuration files:
svn checkout http://svn.osgeo.org/osgeo/foss4g/benchmarking foss4g_benchmark

Some scripts need base path /opt/benchmarking
sudo ln -s $(pwd)/foss4g_benchmark /opt/benchmarking

Download the test data sets:
mkdir data
cd data
#Download data: http://wiki.osgeo.org/wiki/Benchmarking_2009#Download
unzip GNIS-2009.zip
unzip raster-data.zip
unzip vector-data-tiger08-tx-merged.zip
unzip vector-data-tiger08-tx-counties.zip

==== PostgreSQL 8.4 ====

sudo apt-get install postgresql-8.4 postgresql-8.4-postgis

sudo su postgres

psql -c "CREATE ROLE benchmark LOGIN PASSWORD 'benchmark'"

Optional: For local access via socket insert the following line in
/etc/postgresql/8.4/main/pg_hba.conf:
local benchmark benchmark trust

createdb -E latin1 --owner benchmark benchmark
createlang -d benchmark plpgsql
psql -q -d benchmark -f /usr/share/postgresql/8.4/contrib/postgis.sql
psql -q -d benchmark -f /usr/share/postgresql/8.4/contrib/spatial_ref_sys.sql
psql -q -d benchmark -c "GRANT SELECT, UPDATE, INSERT, DELETE ON TABLE public.geometry_columns TO benchmark;"
psql -q -d benchmark -c "GRANT SELECT, UPDATE, INSERT, DELETE ON TABLE public.spatial_ref_sys TO benchmark;"

export PGUSER=benchmark
export PGPASSWORD=benchmark

shp2pgsql -I -D GNIS-2009/gnis_names09.shp gnis_names09 | psql -q -h localhost -d benchmark
shp2pgsql -I -D TIGER-2008/48_TEXAS/areawater_merge.shp areawater_merge | psql -q -h localhost -d benchmark
shp2pgsql -I -D TIGER-2008/48_TEXAS/edges_merge.shp edges_merge | psql -q -h localhost -d benchmark

exit #back to regular user

==== UMN Mapserver ====

sudo apt-get install mapserver-bin cgi-mapserver

Apache Fast-CGI:

apt-get install libapache2-mod-fcgid
ln -s mapserv /usr/lib/cgi-bin/mapserv.fcgi

/etc/apache2/conf.d/fcgi:

<IfModule fcgid_module>
#fcgi settings (see http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html)
DefaultMaxClassProcessCount 8
</IfModule>

/etc/init.d/apache2 reload

==== Installation checks ====

cd foss4g_benchmark/scripts

Basic Mapserver installation check:
shp2img -m ../mapserver/shapefile-merged.map -o /tmp/out.png

Mapserver/Shapefile CGI:
LANG=en_US jmeter -t mapserver/vector/shp_areawater_merge.jmx

==== Setup connection settings ====

Set your connection parameters:
WMS_HOST="localhost"
PG_CONN="dbname=benchmark user=benchmark password=benchmark port=5432 host=localhost"

Replace connection settings in map files:
sed --in-place -e "s/dbname=benchmark user=postgres password=postgres port=5432 host=192.168.0.8/$PG_CONN/" ../mapserver/postgis.map

Replace WMS host name in JMeter configuration:
FILES=$(find . -name '*.jmx')
sed --in-place -e "s/192.168.0.3/$WMS_HOST/" $FILES

Set all jmeter configurations to Mapserver FCGI:
FILES=$(find . -name '*.jmx')
sed --in-place -e "s/mapserv560beta3/mapserv/" $FILES
sed --in-place -e "s/mapserv560beta3.fcgi/mapserv.fcgi/" $FILES
FILES=$(find . -name 'shp_*.jmx')
sed --in-place -e "s!cgi-bin/mapserv<!cgi-bin/mapserv.fcgi<!" $FILES

Set LANG when running jmeter:
sed --in-place -e 's!$JMETER_HOME/jmeter!LANG=en_US jmeter!' benchmark

==== Testing ====

cd foss4g_benchmark/scripts

Mapserver/Shapefile FCGI:
./benchmark mapserver/vector/shp_areawater_merge.jmx

Mapserver/PostgreSQL FCGI:
./benchmark mapserver/vector/pg_fcgi_areawater_merge.jmx

Run all Mapserver tests:
sh mapservconf | tee mapservconf.log
grep -v "Generate" mapservconf.log | grep -v "Waiting"| grep -v "Starting" | grep -v "Created" | grep -v "Tidying" | grep -v "end of run" >mapservconf-summarized.log

== Discusstion ==

Please edit the wiki for improvements and add instructions for your platform.

Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

----

[[Category:FOSS4G]]

@@ Line 12: / Line 12: @@
 * [[Benchmarking 2013 | FOSS4G 2013 Nottingham]]
 * [[Benchmarking_2010 | FOSS4G 2010 Barcelona]]
 * [[Benchmarking_2009 | FOSS4G 2009 Sydney]]

@@ Line 118: / Line 118: @@
    FILES=$(find . -name '*.jmx')
    sed --in-place -e "s/192.168.0.3/$WMS_HOST/" $FILES
 Set all jmeter configurations to Mapserver FCGI:

@@ Line 24: / Line 24: @@
 The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).
-=== Ubuntu 9.10 "Karmic Koala" ===
+=== Running Benchmark 2009 on Ubuntu 9.10 "Karmic Koala" ===
 ==== Install scripts and data sets ====
@@ Line 34: / Line 34: @@
 Get the test scripts and configuration files:
-   svn checkout http://svn.osgeo.org/osgeo/foss4g/benchmarking foss4g_benchmark
+   svn export -r 3084 http://svn.osgeo.org/osgeo/foss4g/benchmarking foss4g_benchmark
 Some scripts need base path /opt/benchmarking
@@ Line 43: / Line 43: @@
    cd data
    #Download data: http://wiki.osgeo.org/wiki/Benchmarking_2009#Download
-   unzip GNIS-2009.zip
+   for i in GNIS-2009.zip raster-data.zip vector-data-tiger08-tx-merged.zip vector-data-tiger08-tx-counties.zip; do
-  unzip raster-data.zip
+    wget --user=foss4g --password=foss4g http://www.maptools.org/foss4g/$i
-  unzip vector-data-tiger08-tx-merged.zip
+    unzip $i
-  unzip vector-data-tiger08-tx-counties.zip
+  done
 ==== PostgreSQL 8.4 ====

@@ Line 19: / Line 19: @@
 The FOSS4G is a useful test suite for your own WMS server installation. This section gives you step-by-step instructions for running the FOSS4G WMS benchmark on different platforms.
 === Ubuntu 9.10 "Karmic Koala" ===
@@ Line 137: / Line 141: @@
    grep -v "Generate" mapservconf.log | grep -v "Waiting"| grep -v "Starting" | grep -v "Created" | grep -v "Tidying" | grep -v "end of run" >mapservconf-summarized.log
-== Discusstion ==
+== Discussion ==
 Please edit the wiki for improvements and add instructions for your platform.