Migration Documentation

2007-01-31T21:03:35Z

Wiki-Sbarnes:

SAC

2007-01-31T21:02:54Z

Wiki-Sbarnes: /* Migration to new Infrastructure */

The OSGeo System Administration Committee (SAC) manages all non-CollabNet computer systems operated by OSGeo. Responsibilities include:

* Develop published policies and procedures for managing systems, and system services.
* Solicit input from stakeholders on requred system services, and to prioritize which to offer and when.
* Assign appropriate security and other responsibilities to administrators and users for access to systems.
* Ensure core OSGeo system services are well maintained, highly available and recoverable in case of system failure.
* Recommend system expenditures to the Board.

= Members =

* Norman Vine (Chair)
* Howard Butler
* John Graham
* Frank Warmerdam
* Arnulf Christl
* Daniel Morissette

= Meetings =
* 1/22/07 [[SAC:1_22_07_agenda]] [[SAC:1_25_07_minutes]]

= Policy Documents =

* [[SAC:Security Groups Policy]] (approved)

= Procedures =
* [[SAC:Admin and Troubleshooting]]
* [[SAC:Create New User]]
* [[SAC:Setup LDAP Authentication]]
* [[SAC:Standard System Setup]]
* [[SAC:ISO Mirroring]]
* [[SAC:Mailing Lists]]

= Resources =

== Documents ==
* [https://sac.osgeo.org/SAC_Motion.html Motion forming SAC]
* [[Infrastructure Transition Plan]]

== Setup ==
* [[SAC TODO List]]
* [[SAC Service Status]] indicates what services are setup, where and known issues.
* [[SAC DNS Registry]] lists DNS entries within the osgeo.org domain.
* [[BuildBot Configuration]]
* [[Download Server]]
* [[Proposed LDAP Layout]]
* [[Community Mapbuilder Continuum Builds]]

== Portal and Communication ==
* [[Drupal Portal]] - transition notes for www project, talks about issues transitioning to new web site portal.
* Mailing list: [https://sac.osgeo.org/servlets/ProjectMailingListList Subscribe] and [https://sac.osgeo.org/servlets/SummarizeList?listName=sac Archive]

== Projects with SAC involved ==
* [[Geodata processing at telascience.org]] - by [[Geodata Packaging Working Group]]
* The [[Infrastructure Working Group]] has some thoughts on services required from SAC and may provide additional input in the future.
* The [[Software Stack]] page talks about the ''Reference Stack'' of software we would like to setup for demonstration and testing purposes.
* Teleconferencing http://openwengo.org/

== Migration to new Infrastructure ==
* The [[Project Infrastructure Migration]] page talks about migration issues of the projects in incubation and provides hints to some services we should be trying to provide.
* [[Migration Tasks and Milestones]]
* [[Migration Documentation]]
[[Category:Infrastructure]]

User:Sbarnes

2007-01-31T20:36:54Z

Wiki-Sbarnes:

== Sandbox Page for osgeo.org transition docs ==

= Description of current layout and installed software =

Notes from Shawn on our PEER1 system:

* OS Version: Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
* For most elements of system i tried to stay with the default redhat locations and red hat el 4 rpms installed from peer 1's up2date repository. Reasoning, the servers are updated automatically against this repository and supported by PEER1 - reduce sysadmin load on keeping packages updated by using packages not available through PEER1 repository

== Installed software using up2date ==
* apache - httpd-2.0.52-28.ent.i386
* postfix - postfix-2.2.10-1.RHEL4.2.i386
* php - php-4.3.9-3.22PIDH.i386
* python - python-2.3.4-14.3.i386
* mailman - mailman-2.1.5.1-34.rhel4.5.i386

== Red Hat EL 4 rpms installed manually (rpm -i) ==
* MySQL-client-standard-5.0.27-0.rhel4.i386.rpm
* MySQL-server-standard-5.0.27-0.rhel4.i386.rpm
* MySQL-devel-standard-5.0.27-0.rhel4.i386.rpm
* MySQL-shared-compat-5.0.27-0.rhel4.i386.rpm
* clearsilver-0.10.1-1.2.el4.rf.i386.rpm
* sqlite-2.8.16-1.2.el4.rf.i386.rpm
* python-clearsilver-0.10.1-1.2.el4.rf.i386.rpm
* python-sqlite-1.0.1-12.el4.rf.i386.rpm
* subversion-1.4.3-0.1.el4.rf.i386.rpm
* mod_dav_svn-1.4.3-0.1.el4.rf.i386.rpm

== Not specific to Red Hat EL 4 rpms installed manually ==
* MySQL-zrm-1.1.2-1.noarch.rpm

== Source Install ==
* drupal-4.7.4.tar.gz
* phpldapadmin-0.9.8.3.tar.gz
* trac-0.10.3.tar.gz

= Paths to services directories =

'''Apache root directory'''
/var/www/html/

'''Subversion parent directory'''
/var/www/svn/repos/

'''Trac parent directory'''
/var/www/trac/

= IP tables custom =

All iptables rules must be written in /etc/sysconfig/iptables-custom

Current custom rules are:

/etc/sysconfig/iptables-custom/SSH
*filter
-A OUTPUT -p tcp --dport 22 -j ACCEPT
COMMIT

/etc/sysconfig/iptables-custom/VERITAS

''note: this file was placed here by PEER1''
*filter
-A INPUT -s 10.0.48.0/24 -d 0/0 -p tcp -j ACCEPT
-A OUTPUT -s 0/0 -d 10.0.48.0/24 -p tcp -j ACCEPT
COMMIT

= System login and maintenance procedures =

'''Note: if change root password must let PEER1 know the new password for backup/restore and tickets. i would refer no root ssh login but, PEER1 admins need root access.'''

= Subversion maintanence =
[http://svnbook.red-bean.com/nightly/en/index.html Subversion Book]
=== Creating a repository ===
$ sudo mkdir /var/www/svn/repos/<repo_name>
$ sudo svnadmin /var/www/svn/repos/<repo_name>
$ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

=== Loading a repository from a dumpfile ===
$ sudo svnadmin load /var/www/svn/repos/<repo_name> < /path/to/dumpfile
$ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

=== Dumping a repository ===
$ sudo svnadmin dump /var/www/svn/repos/<repo_name> > dumpfile

=== Recovering / unlocking repository ===
* subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
* Need policy on who to contact and who can run 'svnadmin recover' as
priviledged access is needed (may need to stop/start apache to drop
requests to repository before recover)

Subversion was upgraded to version 1.4.3 (30 Jan. 2007) to reduce the repository locking problem if ctl-c used to end a checkout.

If the repository locks the following command should be used to recover the repository:

$ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
$ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

In most cases this will work in the odd case that you are waiting a very long time for the command to run then apache may have to be restarted to drop anything accessing the repository and preventing the repository from being recovered

$ sudo /sbin/service httpd restart
$ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
$ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

= TRAC Maintenance =
[http://trac.edgewall.org/wiki/TracGuide Trac Guide]
=== Creating a trac instance ===
$ sudo trac-admin /var/www/trac/<proj_name> initenv
$ sudo chown -R apache:apache /var/www/trac/<proj_name>

=== Hotcopy a trac instance ===
$ sudo trac-admin /var/www/html/trac/<proj_name> hotcopy /path/to/copy/trac/to

= Mailman Maintenance =
=== create a new mailing list ===
$sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
=== renaming a list ===
* create new list
$sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
* move original lists archive to newlists archive location
$ sudo cp /var/lib/mailman/oldlist/oldlist.mbox/oldlist.mbox \
/var/lib/mailman/newlist/newlist.mbox/
* create archive
$ sudo /usr/lib/mailman/bin/arch --wipe newlist
* export subscribers from old list regular and digest members
$ sudo /usr/lib/mailman/bin/list_members -r listname > listname-regular.txt
$ sudo /usr/lib/mailman/bin/list_members -d listname > listname-digest.txt
* import subscribers into new list
$ sudo /usr/lib/mailman/bin/add_members --regular-members=listname-regular.txt --welcome-msg=y listname
$ sudo /usr/lib/mailman/bin/add_members --digest-members=listname-digest.txt --welcome-msg=y listname
* remove old list
$ sudo /usr/lib/mailman/bin/rmlist listname
* update aliases and check that proper permissions are set
$ sudo /usr/lib/mailman/bin/genaliases
$ sudo /usr/lib/mailman/bin/check_perms -f

= Backups =

=== Daily tape backup ===
* PEER1 is doing daily tape backup of entire system

=== Other backups ===
==== Mysql ====
MySQL is being backed up using [http://mysqlbackup.zmanda.com MySQL-zrm]
==== current cron jobs for backups ====
/etc/cron.d/backup.cron

#### Backup cron jobs
# min hour day month dayofweek user command
#
# Daily rsync to /home/back
# /etc
05 10 * * * root /usr/bin/rsync -a --delete /etc/ /home/back/etc/
# /var/www/html
08 10 * * * root /usr/bin/rsync -a --delete /var/www/html/ /home/back/html/
# Every 3 hours
# /var/lib/mailman
20 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mailman/ /home/back/mailman/
# /etc/mysql-zrm
33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
# /var/lib/mysql-zrm
34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
# call backup_trac_svn.sh to backup
# subversion /var/www/svn/repos
# trac /var/www/trac
45 */3 * * * root /root/scripts/backup_trac_svn.sh
# once a day rsync /home/back to osgeo2.osgeo.net
0 22 * * * root /root/scripts/rsync_back.pl

backup script are in /root/scripts

'''backup_trac_svn.sh''' calls separate scripts to backup trac and svn repositories and place tgz files of the backups in /home/back/svn_backup and /home/back/trac_backup which are held for 14 days.
'''rsync_back''' rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)

= FDO Repository Merge =

To merge repositories you need to parse through the dumpfiles.

In the fdo merge all subprojects had the same directory strucuture,
/trunk/Providers/
All dumpfiles had to be 'filtered' to just pull the /trunk/Providers/* and drop branches and tags.

SVNDUMPFILTER

$ svndumpfilter include --help
Filter out nodes without given prefixes from dumpstream
Usage: svndumpfilter include PATH_PREFIX
Options:
--drop-empty-revs Remove revisions emptied by filtering
--renumber-revs Renumbe revisions left after filtering
--preserve-revprops Don't filter revision properties
--quiet Do not display filtering statistics

$ svndumpfilter exclude --help
Filter out nodes with given prefixes from dumpstream
Usage: svndumpfilter exclude PATH_PREFIX
Options:
--drop-empty-revs Remove revisions emptied by filtering
--renumber-revs Renumbe revisions left after filtering
--preserve-revprops Don't filter revision properties
--quiet Do not display filtering statistics

This process followed for all repositories

'''merge fdogdal trunk'''

$ cat fdogdal.dmp | svndumpfilter include trunk/www/ > fdogdal-merge-trunk.www.dmp
$ cat fdogdal.dmp | svndumpfilter include trunk/Providers/ > fdogdal-merge-trunk.Providers.GDAL.dmp

edit file and remove lines,
Node-path: trunk/
....
Node-action: add
....
PROPS-END

Node-path: trunk/www/
....
Node-action: add
....
PROPS-END

Node-path: trunk/Providers/
....
Node-action: add
....
PROPS-END

Rename index.html to fdogdal-index.html
$ perl -pi.bak -e 's/^Node-path:\ trunk\/www\/index.html/Node-path:\ trunk\/www\/fdogdal-index.html/g' fdo-merge.trunk.dmp

$ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp

User:Sbarnes

2007-01-29T19:20:57Z

Wiki-Sbarnes: /* Sandbox Page for osgeo.org transition docs */

User:Sbarnes

2007-01-29T15:53:07Z

Wiki-Sbarnes:

== Sandbox Page for osgeo.org transition docs ==

Migration Tasks and Milestones

2006-11-24T14:20:27Z

Wiki-Sbarnes: /* Install and Configure Key Services [12/08/06] */

This page is intended to describe the tasks and milestones for the Infrastructure Migration project. We'll also use this to track status as we move toward our end of December deadline.

== Servers online at PEER1 [12/01/06] ==
* Negotiate and Sign Contract with PEER1 [Tyler]

== Arrange Dump of CollabNet Databases [12/01/06] ==
Arrange with CollabNet (Geoff)

High priority:
User database,
Maillist and archives,
SVN including history

Low priority:
Groups

== Install and Configure Key Services [12/08/06] ==
* LAMP [Shawn]
* LDAP [Shawn]
* Drupal [Shawn]
* postfix(mta) / MailMan [Shawn]
* SVN, Trac [Shawn]

== Migrate Services and Content [12/20/06] ==
* Develop Scripts to Migrate CollabNet Users to LDAP [Shawn]
* Develop Scripts to Migrate CollabNet Mailing Lists to MailMan [Shawn]
* Develop Scripts for SVN Dump/Load [Shawn]
* Develop Basic Drupal Navigation (e.g. Clone Existing) [Tyler to Coordinate]
* Migrate OSGeo Site Content [Tyler to Coordinate]
* Migrate MapGuide Project Content [Bob to Coordinate]
* Migrate FDO Project Content [Bob to Coordinate]

== Go Live Phase 1 - Basic Site [12/22/06] ==
* Run Migration Scripts [Shawn]
* Change DNS for osgeo.org [Shawn]

At this point the OSGeo Web Site and Mailing lists will be up and running on the new infrastructure.

== Go Live Phase 2 - SVN [12/29/06] ==
* Migrate SVN Repositories (MapGuide, FDO, Mapbender) [Shawn]
* Extract Project Tracker Artifacts e.g. Bugs as XML (MapGuide and FDO) [Shawn]

At this point the SVN instances for MapGuide, FDO, and Mapbender will be live on the new infrastructure.

== Go Live Phase 3 - Trac and Cleanup [01/31/07] ==
* Develop scripts to import Project Tracker Artifacts into Trac [Shawn]
* Cleanup Migration Issues [Shawn, All]
* Develop Long Term Drupal Navigation, Organization, Etc. [Tyler to Coordinate]

== Risks ==

CollabNet database dumps

Migrating user databases to LDAP

Configuring LDAP