OSGeo - User contributions [en]

Benchmarking 2009

2009-10-12T06:00:33Z

Wiki-Anakintang: /* ArcGIS Server */

== Overview ==

{| border="1"
|Description
|The performance shoot-out will see mapping servers compared in terms of how long they take to generate a map image, from common set of spatial data.
|-
|Participants
|development teams from ArcGIS Server, GeoServer, MapServer
|-
|Results Announced
|[http://2009.foss4g.org/ FOSS4G2009] event, 20-23 October, 2009, in Sydney Australia
|-
|Press Release
|[[FOSS4G_2009_Press_Release_32]]

|}

== Participants ==

* Geoserver
** Andrea Aime
** [[User:simboss | Simone Giannecchini]]
* MapServer
** [[User:Dmorissette | Daniel Morissette]]
** [[User:jmckenna | Jeff McKenna]]
** Paul Ramsey
** Frank Warmerdam (raster)
* PostGIS
** Paul Ramsey
* Oracle
** Mike Smith
* SDE
** Mike Smith
* ArcGIS Server
** Satish Sankaran

== Basic Premise ==

The performance shoot-out presentation at [[FOSS4G2009]] will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.

== Rules of Engagement ==

# All parties must contribute any changes that they make to their software for this exercise, back to their community.
# Comparisons will be made of the best available version of the software, be it a formal release or a development version. For example, MapServer 5.6.0 (betas) will be tested, while GeoServer 2.0 (dev) will be tested.
# Data formats used must be shapefiles for vectors, geotiff files for rasters, and PostgreSQL/PostGIS for database. If you wish to test other formats/backends (such as SDE or ECW) you may do so, but that is optional for all parties. See below for access to the required data.
# Output formats are: jpeg for imagery, gif for non-antialised vector provided we do any, and png24 for antialiased
# Nearest Neighbor resampling should be used for raster image display

== Coordination ==

Coordination is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net

=== Next IRC meeting ===

* Wed Oct 6th: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=10&day=6&hour=15&min=0&sec=0
** Agenda:
***

=== Previous Meetings ===

* Wed Sept 23rd: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=23&hour=15&min=0&sec=0
** Agenda:
*** moving scripts/files into SVN on BenchmarkingA server
*** LD_LIBRARY_PATH issue on BenchmarkingA server
*** FYI data download available
*** new vector data available on BenchmarkingA server (/opt/data/TIGER-2008)
**** what layers to use
*** what scales should be styled
**** e.g. MapServer scale of [http://labs.gatewaygeomatics.com/cgi-bin/wms_benchmarking_by_merged?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=vector_benchmarking&STYLES=&SRS=EPSG:4269&BBOX=-97.038385,32.545214,-96.516866,32.989369&WIDTH=950&HEIGHT=600&FORMAT=image/png 1:200,000 of Dallas]
**** e.g. MapServer scale of [http://labs.gatewaygeomatics.com/cgi-bin/wms_benchmarking_by_merged?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=vector_benchmarking&STYLES=&SRS=EPSG:4269&BBOX=-97.0286,32.5511,-96.8724,32.6925&WIDTH=950&HEIGHT=600&FORMAT=image/png 1:75,000 of Duncanville/Dallas] (with some ESRI icons)
* Wed Sept 16th: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=16&hour=15&min=0&sec=0
** Agenda:
*** Directory structure in /opt/ ?
*** Status of ERDAS
*** status of home for the benchmarking files: http://trac.osgeo.org/osgeo/ticket/435
*** feedback on vector styling
** Minutes
*** msmith: will setup SDE+Oracle on RedHat by Tues
*** augusttown: will draft gnis_pop & tiger_trac styling
*** jmckenna: possibly have maptools.org host the benchmarking files?
*** frankw: press release should for now include GeoServer, MapServer, ESRI, and 'other servers' (until ERDAS confirms)

* Wed Sept 9th: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=09&hour=15&min=0&sec=0
** Minutes:
*** jmckenna: Need confirmation from ERDAS so press release can move forward
*** msmith: can setup rhel 5 64 bit with sde 9.3.1 and oracle 11g 64 bit for SDE (need confirmation from ESRI that this setup is OK)
*** aaime: need feedback on styling (icon for point styling, for example)
*** aaime: all should use mailing list more often, for feedback
*** aaime: proposed roads style on mailing list
*** augusttown: are we ok with having a separate machine for database?
*** aaime: postgres should be on that same machine (jmckenna agrees)
*** augusttown: please post possible operating system options for database machine and we will get back to you
*** msmith: pramsey do you want to install postgres/postgis on new db machine?
*** jmckenna: setup an OSGeo server for downloading benchmarking data ([http://trac.osgeo.org/osgeo/ticket/435 ticket])
*** msmith: local port forwarding is ok for doing the testing or do I need to set up the NAT to handle some of the web traffic out?

* Wed Sept 2nd: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=02&hour=14&min=0&sec=0
** Agenda:
*** update on ERDAS involvement
*** what other web server teams have been contacted? Should more be contacted?
*** work on a press release: http://lists.osgeo.org/pipermail/benchmarking/2009-August/000107.html
*** output format (jpeg, png 24 bit, png 8 bit, gif?)
*** antialiasing (enabled, or not?)
*** interpolation (nearest neighbour, bilinear, bicubic?)
*** specifics on raster formats tests, especially the TIFF part (multilayers, mosaicked, ...)

* Wed Aug 26
** discussion logs: http://logs.qgis.org/foss4g/%23foss4g.2009-08-26.log
** summary:
*** Frank: to confirm with ESRI about Operating System (RHEL or CentOS)
*** Mike: to update wiki with specs when OS switch is done (if required)
*** discussion on how to test (OGC versus native access)...WMS was agreed upon
*** Daniel: volunteers to look at MapServer's WMS vs CGI performance (which is optional)
*** Data: we'll use same data as previous years (but plan to update data to next TIGER or OSM release next year)
**** all agreed to use shapefiles, geotiffs, and PostGIS. to be verified with ESRI.
**** Jeff: to review Andrea's progress re: layer stying
*** foss4g2009: Andrea and Jeff will co-present. (plus an ESRI representative??)
**** press release needs to be drafted: http://lists.osgeo.org/pipermail/benchmarking/2009-August/000107.html

== Hardware ==

Machine A (server)
* System Type: Dell PowerEdge 2850
* Ship Date: 10/21/2004
* 2 X Processor, 80546K, 3.4G, 1M, Xeon Nocona, 800
* 4 X DIMM, 2G 400M, 128X72, 8, 240, 2RX4
* 6 X Hard Drive, 73GB, SCSI, U320, 15K
* Service Tag: 16GRV51
* OS: Centos 5.3 + updates

Machine B (testing box)
* System Type: Dell PowerEdge 1750
* Ship Date: 9/23/2003
* 1 x Processor, 80532K, 3.06GHz, 512K 533
* 2 x Dimm, 512, 266M, 64X72, 8K, 184, 1U
* 2 x Hard Drive, 300Gb, SCSI, U320, 10K
* OS: Centos 5.3 + updates

== Layers ==

* '''[[texas_roads_styled]]''' This test exercises the renderer a bit more. The lines will be "pipe-styled" and different road types will have different styles. ''Data: edges_merge''
* '''[[point_layer_styled]]''' This test exercises the ability of the renderer to build a map in which points are symbolized with a certain number of externally provided icons (png/svg) depending on some point attribute. ''Data: gnis_names09''
* '''[[polygon_layer_styled]]''' Tests polygon rendering, with a solid fill. For this test the type of classification matters less than the number and complexity of the polygons (number of vertices and rings) which is the aspect we should work on. e.g. render maps with a few complex polygons vs other maps with lots of small polygons. ''Data: areawater_merge''
* '''single big ECW file''' (actual data TBD). This test exercises the ability to read a single wavelet compressed file and return it in JPEG format.
* '''file system mosaic of TIFF tiles'''. Generated out of the ECW file by splitting it into a sizeable amount of tiles, this test checks the ability of the server to efficiently retrieve data from a large collection of images and deal the associated file handling issues (ulimit and the like). It can also serve as a comparison with ECW results.

== Data ==

Data will reside in '''/opt/data''' on the testing server. Do not add any data to this directory without also describing it fully on this page.

* Vector data (original):
** '''gnis_names.shp''' (POINT, EPSG:4326) US named feature points.
** '''states.shp''' (POLYGON, EPSG:4326) US states and demographics.
** '''tiger_shp.shp''' (LINESTRING, EPSG:4326) TIGER roads for Texas.
** '''tiger_tracts.shp''' (POLYGON, EPSG:4326) TIGER census tracts for USA.
* Vector data (GNIS Names 2009)
** '''/GNIS-2009/gnis_names09.shp''' (POINT, EPSG:4269) US named feature points, for 2009.
* Vector data (TIGER 2008 of Texas)
** separated into individual counties (like the TIGER release)
*** '''/TIGER-2008/48_TEXAS/County/arealm.shp''' (POLYGON, EPSG:4269) area landmarks (e.g. parks) for counties.
*** '''/TIGER-2008/48_TEXAS/County/areawater.shp''' (POLYGON, EPSG:4269) area water(e.g. lakes) for counties.
*** '''/TIGER-2008/48_TEXAS/County/edges.shp''' (LINE, EPSG:4269) linework (e.g. roads, rivers) for counties.
*** '''/TIGER-2008/48_TEXAS/County/pointlm.shp''' (POINT, EPSG:4269) point landmarks (e.g. hospital, airport) for counties.
*** '''/TIGER-2008/48_TEXAS/county_tileindex.shp''' (POLYGON, EPSG:4269) county index file used by MapServer.
** merged into single files for the entire state (required for servers that cannot load many .shp files as a single entity)
*** '''/TIGER-2008/48_TEXAS/arealm_merge.shp''' (POLYGON, EPSG:4269) area landmarks (e.g. parks) for entire state.
*** '''/TIGER-2008/48_TEXAS/areawater_merge.shp''' (POLYGON, EPSG:4269) area water(e.g. lakes) for entire state.
*** '''/TIGER-2008/48_TEXAS/edges_merge.shp''' (LINE, EPSG:4269) linework (e.g. roads, rivers) for entire state.
*** '''/TIGER-2008/48_TEXAS/pointlm_merge.shp''' (POINT, EPSG:4269) point landmarks (e.g. hospital, airport) for entire state.
** general
*** '''/TIGER-2008/48_TEXAS/tl_2008_48_place.shp''' (POLYGON, EPSG:4269) places (populated areas) for entire state.
*** '''/TIGER-2008/tl_2008_us_county.shp''' (POLYGON, EPSG:4269) county outlines for entire state.
*** '''/TIGER-2008/tl_2008_us_state.shp''' (POLYGON, EPSG:4269) US state outlines.
* Raster data
** '''world-topo-bathy-200409-3x86400x43200.ecw''' (ECW, EPSG:4326) Medium size ECW file (Bluemarble Next Generation, whole world)
*** '''world-topo-bathy-200409-3x86400x43200.tif''' The same as a tiled BigTIFF, with overviews.
*** '''bluemarble_512/*.tif''' The same split into 512 GeoTIFF tiles, each with overviews.
* PostGIS data
** dbname=benchmark user=postgres password=postgres port=5432 host=localhost
** tables: public.gnis_names public.states public.tiger_shp public.tiger_tracts public.edges_merge public.pointlm_merge public.arealm_merge public.areawater_merge public.gnis_names09
** geometry column: "the_geom" in all cases
** srid: 4326 in all cases

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** raster-data.zip (.4 GB)
** vector-data-original.zip (.5 GB) (old tiger data, merged, for Texas)
** vector-data-tiger08-tx-counties.zip (1.1 GB) (TIGER 08, stored by county, for Texas)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

== Live BenchmarkA WMS GetMap Requests ==

=== MapServer ===

* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged,roads-merged,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png PostGIS]

* [http://64.222.187.168/cgi-bin/mapserv560beta3.fcgi?MAP=/opt/benchmarking/mapserver/postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged,roads-merged,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png PostGIS fcgi]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/shapefile-merged.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged,roads-merged,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Merged Shapefiles]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/shapefile-merged.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=roads-merged&STYLES=&SRS=EPSG:4269&WIDTH=950&HEIGHT=600&FORMAT=image/png&BBOX=-96.79344550384396,32.76888304968294,-96.78588564285151,32.77365769873079 Merged Shapefiles - Local Roads labelled]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/shapefile-county.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-county,roads-county,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png County-based Shapefiles]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-oraclespatial,roads-merged-oraclespatial,gnis-names-oraclespatial&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png OracleSpatial]
* [http://64.222.187.168/cgi-bin/mapserv560beta3.fcgi?MAP=/opt/benchmarking/mapserver/oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-oraclespatial,roads-merged-oraclespatial,gnis-names-oraclespatial&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png OracleSpatial fcgi]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-oracle-ogr,roads-merged-oracle-ogr,gnis-names-oracle-ogr&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Oracle through OGR]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/sde.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-sde,roads-merged-sde,gnis-names-sde&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE]
* [http://64.222.187.168/cgi-bin/mapserv560beta3.fcgi?MAP=/opt/benchmarking/mapserver/sde.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-sde,roads-merged-sde,gnis-names-sde&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE fcgi]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/sde.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-sde-ogr,roads-merged-sde-ogr,gnis-names-sde-ogr&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE through OGR]

=== GeoServer ===

* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=postgis&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Postgis group]
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=shapefiles&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Shapefiles group]
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=shp:edges_merge&STYLES=roads_classified_shp_label&SRS=EPSG:4269&WIDTH=950&HEIGHT=600&FORMAT=image/png&BBOX=-96.79344550384396,32.76888304968294,-96.78588564285151,32.77365769873079 Shapefiles, labelled]
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=oracle&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Oracle group]
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=sde&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE group]

== Software ==

For those of us who have to build our software, the builds will be staged in '''/opt/build'''

=== Libraries ===

* GEOS 3.1.1 installed in /usr/local (pramsey)
* Proj4 SVN trunk installed in /usr/local (pramsey)
* libecw installed in /usr/local (pramsey)
* GDAL 1.6.1 installed in /usr/local (pramsey)
* AGG 2.5 installed in /usr/local (pramsey)
* FastCGI installed in /usr/local (pramsey)

=== Databases ===

* PostgreSQL installed in /usr/local/pgsql (pramsey)
** PostgreSQL 8.3.7
** PostGIS 1.4.0
** PGDATA in /home/postgres/data

=== GeoServer related ===
* Sun JDK 1.6.0_16 with JAI, JAI Image-IO and ImageIO-ext native extensions installed in (/opt/jdk1.6.0_16) (aaime)
* GeoServer binaries and configurations (/opt/geoserver) (still incomplete at the moment) (aaime)

=== Testing machine ===

* Plain Sun JDK 1.6.0_16 (/opt/jdk1.6.0_16) (aaime)
* JMeter (/opt/jakarta-jmeter-2.3.4) (aaime)
* symbolic link to start jmeter in /usr/local/bin/jmeter (instructions on how to use it will follow) (aaime)
* /usr/local/bin/wms_requests.py, simple tool to generate random bounding boxes and sizes to drive JMeter (aaime, tool developed by Frank W.)

=== Starting / Stopping Services ===

To shutdown apache, and associated mapserver fastcgi processes:

sudo /usr/sbin/apachectl stop

For MapServer team: you might want to do a graceful restart of Apache, before running tests

sudo /usr/sbin/apachectl graceful

To shutdown arcgis:

sudo su - arcgis
cd /arcgis/scripts
./stopserver

To shutdown GeoServer do the following

sudo /opt/geoserver/gs20x/catalina.sh stop
ps aux | grep java | grep geoserver

The second command can be used to confirm GeoServer is really down, if not, kill it manually ('kill -9 PID').

== SVN ==

The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).

== Press Release ==

[[FOSS4G_2009_Press_Release_32]]

----

[[Category:FOSS4G2009]]
[[Category:FOSS4G]]

Benchmarking 2009

2009-10-09T17:28:35Z

Wiki-Anakintang: /* Live BenchmarkA WMS GetMap Requests */

== Overview ==

{| border="1"
|Description
|The performance shoot-out will see mapping servers compared in terms of how long they take to generate a map image, from common set of spatial data.
|-
|Participants
|development teams from ArcGIS Server, GeoServer, MapServer
|-
|Results Announced
|[http://2009.foss4g.org/ FOSS4G2009] event, 20-23 October, 2009, in Sydney Australia
|-
|Press Release
|[[FOSS4G_2009_Press_Release_32]]

|}

== Participants ==

* Geoserver
** Andrea Aime
** [[User:simboss | Simone Giannecchini]]
* MapServer
** [[User:Dmorissette | Daniel Morissette]]
** [[User:jmckenna | Jeff McKenna]]
** Paul Ramsey
** Frank Warmerdam (raster)
* PostGIS
** Paul Ramsey
* Oracle
** Mike Smith
* SDE
** Mike Smith
* ArcGIS Server
** Satish Sankaran

== Basic Premise ==

The performance shoot-out presentation at [[FOSS4G2009]] will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.

== Rules of Engagement ==

# All parties must contribute any changes that they make to their software for this exercise, back to their community.
# Comparisons will be made of the best available version of the software, be it a formal release or a development version. For example, MapServer 5.6.0 (betas) will be tested, while GeoServer 2.0 (dev) will be tested.
# Data formats used must be shapefiles for vectors, geotiff files for rasters, and PostgreSQL/PostGIS for database. If you wish to test other formats/backends (such as SDE or ECW) you may do so, but that is optional for all parties. See below for access to the required data.
# Output formats are: jpeg for imagery, gif for non-antialised vector provided we do any, and png24 for antialiased
# Nearest Neighbor resampling should be used for raster image display

== Coordination ==

Coordination is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net

=== Next IRC meeting ===

* Wed Oct 6th: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=10&day=6&hour=15&min=0&sec=0
** Agenda:
***

=== Previous Meetings ===

* Wed Sept 23rd: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=23&hour=15&min=0&sec=0
** Agenda:
*** moving scripts/files into SVN on BenchmarkingA server
*** LD_LIBRARY_PATH issue on BenchmarkingA server
*** FYI data download available
*** new vector data available on BenchmarkingA server (/opt/data/TIGER-2008)
**** what layers to use
*** what scales should be styled
**** e.g. MapServer scale of [http://labs.gatewaygeomatics.com/cgi-bin/wms_benchmarking_by_merged?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=vector_benchmarking&STYLES=&SRS=EPSG:4269&BBOX=-97.038385,32.545214,-96.516866,32.989369&WIDTH=950&HEIGHT=600&FORMAT=image/png 1:200,000 of Dallas]
**** e.g. MapServer scale of [http://labs.gatewaygeomatics.com/cgi-bin/wms_benchmarking_by_merged?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=vector_benchmarking&STYLES=&SRS=EPSG:4269&BBOX=-97.0286,32.5511,-96.8724,32.6925&WIDTH=950&HEIGHT=600&FORMAT=image/png 1:75,000 of Duncanville/Dallas] (with some ESRI icons)
* Wed Sept 16th: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=16&hour=15&min=0&sec=0
** Agenda:
*** Directory structure in /opt/ ?
*** Status of ERDAS
*** status of home for the benchmarking files: http://trac.osgeo.org/osgeo/ticket/435
*** feedback on vector styling
** Minutes
*** msmith: will setup SDE+Oracle on RedHat by Tues
*** augusttown: will draft gnis_pop & tiger_trac styling
*** jmckenna: possibly have maptools.org host the benchmarking files?
*** frankw: press release should for now include GeoServer, MapServer, ESRI, and 'other servers' (until ERDAS confirms)

* Wed Sept 9th: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=09&hour=15&min=0&sec=0
** Minutes:
*** jmckenna: Need confirmation from ERDAS so press release can move forward
*** msmith: can setup rhel 5 64 bit with sde 9.3.1 and oracle 11g 64 bit for SDE (need confirmation from ESRI that this setup is OK)
*** aaime: need feedback on styling (icon for point styling, for example)
*** aaime: all should use mailing list more often, for feedback
*** aaime: proposed roads style on mailing list
*** augusttown: are we ok with having a separate machine for database?
*** aaime: postgres should be on that same machine (jmckenna agrees)
*** augusttown: please post possible operating system options for database machine and we will get back to you
*** msmith: pramsey do you want to install postgres/postgis on new db machine?
*** jmckenna: setup an OSGeo server for downloading benchmarking data ([http://trac.osgeo.org/osgeo/ticket/435 ticket])
*** msmith: local port forwarding is ok for doing the testing or do I need to set up the NAT to handle some of the web traffic out?

* Wed Sept 2nd: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=02&hour=14&min=0&sec=0
** Agenda:
*** update on ERDAS involvement
*** what other web server teams have been contacted? Should more be contacted?
*** work on a press release: http://lists.osgeo.org/pipermail/benchmarking/2009-August/000107.html
*** output format (jpeg, png 24 bit, png 8 bit, gif?)
*** antialiasing (enabled, or not?)
*** interpolation (nearest neighbour, bilinear, bicubic?)
*** specifics on raster formats tests, especially the TIFF part (multilayers, mosaicked, ...)

* Wed Aug 26
** discussion logs: http://logs.qgis.org/foss4g/%23foss4g.2009-08-26.log
** summary:
*** Frank: to confirm with ESRI about Operating System (RHEL or CentOS)
*** Mike: to update wiki with specs when OS switch is done (if required)
*** discussion on how to test (OGC versus native access)...WMS was agreed upon
*** Daniel: volunteers to look at MapServer's WMS vs CGI performance (which is optional)
*** Data: we'll use same data as previous years (but plan to update data to next TIGER or OSM release next year)
**** all agreed to use shapefiles, geotiffs, and PostGIS. to be verified with ESRI.
**** Jeff: to review Andrea's progress re: layer stying
*** foss4g2009: Andrea and Jeff will co-present. (plus an ESRI representative??)
**** press release needs to be drafted: http://lists.osgeo.org/pipermail/benchmarking/2009-August/000107.html

== Hardware ==

Machine A (server)
* System Type: Dell PowerEdge 2850
* Ship Date: 10/21/2004
* 2 X Processor, 80546K, 3.4G, 1M, Xeon Nocona, 800
* 4 X DIMM, 2G 400M, 128X72, 8, 240, 2RX4
* 6 X Hard Drive, 73GB, SCSI, U320, 15K
* Service Tag: 16GRV51
* OS: Centos 5.3 + updates

Machine B (testing box)
* System Type: Dell PowerEdge 1750
* Ship Date: 9/23/2003
* 1 x Processor, 80532K, 3.06GHz, 512K 533
* 2 x Dimm, 512, 266M, 64X72, 8K, 184, 1U
* 2 x Hard Drive, 300Gb, SCSI, U320, 10K
* OS: Centos 5.3 + updates

== Layers ==

* '''[[texas_roads_styled]]''' This test exercises the renderer a bit more. The lines will be "pipe-styled" and different road types will have different styles. ''Data: edges_merge''
* '''[[point_layer_styled]]''' This test exercises the ability of the renderer to build a map in which points are symbolized with a certain number of externally provided icons (png/svg) depending on some point attribute. ''Data: gnis_names09''
* '''[[polygon_layer_styled]]''' Tests polygon rendering, with a solid fill. For this test the type of classification matters less than the number and complexity of the polygons (number of vertices and rings) which is the aspect we should work on. e.g. render maps with a few complex polygons vs other maps with lots of small polygons. ''Data: areawater_merge''
* '''single big ECW file''' (actual data TBD). This test exercises the ability to read a single wavelet compressed file and return it in JPEG format.
* '''file system mosaic of TIFF tiles'''. Generated out of the ECW file by splitting it into a sizeable amount of tiles, this test checks the ability of the server to efficiently retrieve data from a large collection of images and deal the associated file handling issues (ulimit and the like). It can also serve as a comparison with ECW results.

== Data ==

Data will reside in '''/opt/data''' on the testing server. Do not add any data to this directory without also describing it fully on this page.

* Vector data (original):
** '''gnis_names.shp''' (POINT, EPSG:4326) US named feature points.
** '''states.shp''' (POLYGON, EPSG:4326) US states and demographics.
** '''tiger_shp.shp''' (LINESTRING, EPSG:4326) TIGER roads for Texas.
** '''tiger_tracts.shp''' (POLYGON, EPSG:4326) TIGER census tracts for USA.
* Vector data (GNIS Names 2009)
** '''/GNIS-2009/gnis_names09.shp''' (POINT, EPSG:4269) US named feature points, for 2009.
* Vector data (TIGER 2008 of Texas)
** separated into individual counties (like the TIGER release)
*** '''/TIGER-2008/48_TEXAS/County/arealm.shp''' (POLYGON, EPSG:4269) area landmarks (e.g. parks) for counties.
*** '''/TIGER-2008/48_TEXAS/County/areawater.shp''' (POLYGON, EPSG:4269) area water(e.g. lakes) for counties.
*** '''/TIGER-2008/48_TEXAS/County/edges.shp''' (LINE, EPSG:4269) linework (e.g. roads, rivers) for counties.
*** '''/TIGER-2008/48_TEXAS/County/pointlm.shp''' (POINT, EPSG:4269) point landmarks (e.g. hospital, airport) for counties.
*** '''/TIGER-2008/48_TEXAS/county_tileindex.shp''' (POLYGON, EPSG:4269) county index file used by MapServer.
** merged into single files for the entire state (required for servers that cannot load many .shp files as a single entity)
*** '''/TIGER-2008/48_TEXAS/arealm_merge.shp''' (POLYGON, EPSG:4269) area landmarks (e.g. parks) for entire state.
*** '''/TIGER-2008/48_TEXAS/areawater_merge.shp''' (POLYGON, EPSG:4269) area water(e.g. lakes) for entire state.
*** '''/TIGER-2008/48_TEXAS/edges_merge.shp''' (LINE, EPSG:4269) linework (e.g. roads, rivers) for entire state.
*** '''/TIGER-2008/48_TEXAS/pointlm_merge.shp''' (POINT, EPSG:4269) point landmarks (e.g. hospital, airport) for entire state.
** general
*** '''/TIGER-2008/48_TEXAS/tl_2008_48_place.shp''' (POLYGON, EPSG:4269) places (populated areas) for entire state.
*** '''/TIGER-2008/tl_2008_us_county.shp''' (POLYGON, EPSG:4269) county outlines for entire state.
*** '''/TIGER-2008/tl_2008_us_state.shp''' (POLYGON, EPSG:4269) US state outlines.
* Raster data
** '''world-topo-bathy-200409-3x86400x43200.ecw''' (ECW, EPSG:4326) Medium size ECW file (Bluemarble Next Generation, whole world)
*** '''world-topo-bathy-200409-3x86400x43200.tif''' The same as a tiled BigTIFF, with overviews.
*** '''bluemarble_512/*.tif''' The same split into 512 GeoTIFF tiles, each with overviews.
* PostGIS data
** dbname=benchmark user=postgres password=postgres port=5432 host=localhost
** tables: public.gnis_names public.states public.tiger_shp public.tiger_tracts public.edges_merge public.pointlm_merge public.arealm_merge public.areawater_merge public.gnis_names09
** geometry column: "the_geom" in all cases
** srid: 4326 in all cases

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** raster-data.zip (.4 GB)
** vector-data-original.zip (.5 GB) (old tiger data, merged, for Texas)
** vector-data-tiger08-tx-counties.zip (1.1 GB) (TIGER 08, stored by county, for Texas)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

== Live BenchmarkA WMS GetMap Requests ==

=== MapServer ===

* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged,roads-merged,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png PostGIS]

* [http://64.222.187.168/cgi-bin/mapserv560beta3.fcgi?MAP=/opt/benchmarking/mapserver/postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged,roads-merged,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png PostGIS fcgi]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/vector-by-merged.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged,roads-merged,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Merged Shapefiles]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/vector-by-merged.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=roads-merged&STYLES=&SRS=EPSG:4269&WIDTH=950&HEIGHT=600&FORMAT=image/png&BBOX=-96.79344550384396,32.76888304968294,-96.78588564285151,32.77365769873079 Merged Shapefiles - Local Roads labeled]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/vector-by-county.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-county,roads-county,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png County-based Shapefiles]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-oraclespatial,roads-merged-oraclespatial,gnis-names-oraclespatial&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png OracleSpatial]
* [http://64.222.187.168/cgi-bin/mapserv560beta3.fcgi?MAP=/opt/benchmarking/mapserver/oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-oraclespatial,roads-merged-oraclespatial,gnis-names-oraclespatial&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png OracleSpatial fcgi]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-oracle-ogr,roads-merged-oracle-ogr,gnis-names-oracle-ogr&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Oracle through OGR]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/sde.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-sde,roads-merged-sde,gnis-names-sde&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE]
* [http://64.222.187.168/cgi-bin/mapserv560beta3.fcgi?MAP=/opt/benchmarking/mapserver/sde.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-sde,roads-merged-sde,gnis-names-sde&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE fcgi]
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/sde.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-sde-ogr,roads-merged-sde-ogr,gnis-names-sde-ogr&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE through OGR]

=== GeoServer ===

* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=postgis&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Postgis group]
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=shapefiles&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Shapefiles group]
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=shp:edges_merge&STYLES=roads_classified_shp_label&SRS=EPSG:4269&WIDTH=950&HEIGHT=600&FORMAT=image/png&BBOX=-96.79344550384396,32.76888304968294,-96.78588564285151,32.77365769873079 Shapefiles, labelled]
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=oracle&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Oracle group]
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=sde&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE group]

=== ArcGIS Server ===

* [http://64.222.187.168:8399/arcgis/services/stacked/MapServer/WMSServer?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=0,1,2&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png&TRANSPARENT=false Shapefile stacked]

* [http://64.222.187.168:8399/arcgis/services/edges_merge_labeled/MapServer/WMSServer?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=0&STYLES=&SRS=EPSG:4269&WIDTH=950&HEIGHT=600&FORMAT=image/png&BBOX=-96.79344550384396,32.76888304968294,-96.78588564285151,32.77365769873079&TRANSPARENT=false Shapefile stacked with label]

== Software ==

For those of us who have to build our software, the builds will be staged in '''/opt/build'''

=== Libraries ===

* GEOS 3.1.1 installed in /usr/local (pramsey)
* Proj4 SVN trunk installed in /usr/local (pramsey)
* libecw installed in /usr/local (pramsey)
* GDAL 1.6.1 installed in /usr/local (pramsey)
* AGG 2.5 installed in /usr/local (pramsey)
* FastCGI installed in /usr/local (pramsey)

=== Databases ===

* PostgreSQL installed in /usr/local/pgsql (pramsey)
** PostgreSQL 8.3.7
** PostGIS 1.4.0
** PGDATA in /home/postgres/data

=== GeoServer related ===
* Sun JDK 1.6.0_16 with JAI, JAI Image-IO and ImageIO-ext native extensions installed in (/opt/jdk1.6.0_16) (aaime)
* GeoServer binaries and configurations (/opt/geoserver) (still incomplete at the moment) (aaime)

=== Testing machine ===

* Plain Sun JDK 1.6.0_16 (/opt/jdk1.6.0_16) (aaime)
* JMeter (/opt/jakarta-jmeter-2.3.4) (aaime)
* symbolic link to start jmeter in /usr/local/bin/jmeter (instructions on how to use it will follow) (aaime)
* /usr/local/bin/wms_requests.py, simple tool to generate random bounding boxes and sizes to drive JMeter (aaime, tool developed by Frank W.)

=== Starting / Stopping Services ===

To shutdown apache, and associated mapserver fastcgi processes:

sudo /usr/sbin/apachectl stop

For MapServer team: you might want to do a graceful restart of Apache, before running tests

sudo /usr/sbin/apachectl graceful

To shutdown arcgis:

sudo su - arcgis
cd /arcgis/scripts
./stopserver

To shutdown GeoServer do the following

sudo /opt/geoserver/gs20x/catalina.sh stop
ps aux | grep java | grep geoserver

The second command can be used to confirm GeoServer is really down, if not, kill it manually ('kill -9 PID').

== SVN ==

The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).

== Press Release ==

[[FOSS4G_2009_Press_Release_32]]

----

[[Category:FOSS4G2009]]
[[Category:FOSS4G]]