This is me, Wu!
» e-shell.org
In this page
Rate this page!
Search inside the wiki!
  Home >> hacks >> Keeping your Zope servers in sync

Keeping your Zope servers in sync

Think about an environment where you have developed a webapp, and you have a copy of the app both in a production server and in a development server. This is an usual approach, because it *lets potential users work with your webapp, while you keep improving it.

Imagine now you have made some big changes to the webapp, and not only existing files are involved in such changes (and therefor changed), but new files (like ZSQL methods) were added. When you proceed with the upgrade of the webapp version running in the production server, upgrading the files one by one from the ZMI could be a mess, and using zope's import/export_ system could not be an option if you can't access your production server by other means than http.

That was my problem, I had limited access to the production server and I needed to speed up updates... Fortunately (again) betabug pointed me to ZSyncer:

ZSyncer is a project that allows live zope objects to be synchronized from one Zope
to another without doing the tedious export / transfer / import dance.
It is very easy to use.

ZSyncer also allows you to see at a glance which objects are out-of-sync.

ZSyncer also allows you to view a diff against the destination object, for many types
of objects.

So, I'll explain how I use Zsyncer to sync two Zope servers, both running under FreeBSD, this are the details of the servers:

Development (local) server:

<wyrmslayer ~ > uname -ap
FreeBSD wyrmslayer.e-shell.org 5.4-STABLE FreeBSD 5.4-STABLE #5: Thu Dec 29 17:27:03 CET 2005     root@wyrmslayer.e-shell.org:/usr/obj/usr/src/sys/WYRMSLAYER  i386 i386
<wyrmslayer ~ > pkg_info|grep zope
zope-2.7.7          An object-based web application platform
<wyrmslayer ~ >

Production (remote) server:

[Frey] ~> uname -ap
FreeBSD Frey.e-shell.org 5.4-STABLE FreeBSD 5.4-STABLE #1: Tue Oct 25 12:29:17 CEST 2005     root@Frey.e-shell.org:/usr/obj/usr/src/sys/FREY  i386 i386
[Frey] ~> pkg_info|grep zope
zope-2.7.6          An object-based web application platform
[Frey] ~>

In both servers Zope was installed via the FreeBSD? ports collection, compiled with the same flags and configured with the same options (just cd /usr/ports/www/zope && make install).

Install Zsyncer is just as easy as installing almost any Zope Product. Just download it, decompress it inside your Zope instance Producs directory and restart your Zope server:

<wyrmslayer /usr/local/www/Zope/e-shell/Products > wget -c http://mesh.dl.sourceforge.net/sourceforge/zsyncer/ZSyncer-0.7.1-beta1.tgz
--13:33:25--  http://mesh.dl.sourceforge.net/sourceforge/zsyncer/ZSyncer-0.7.1-beta1.tgz
           => `ZSyncer-0.7.1-beta1.tgz'
Resolviendo mesh.dl.sourceforge.net... hecho.
Conectando con mesh.dl.sourceforge.net[213.203.218.122]:80... conectado.
PeticiĆ³n HTTP enviada, esperando respuesta... 200 OK
Longitud: 82,904 [application/x-gzip]

100%[======================================================================================>] 82,904       126.70K/s    ETA 00:00

13:33:27 (126.70 KB/s) - `ZSyncer-0.7.1-beta1.tgz' saved [82904/82904]

<wyrmslayer /usr/local/www/Zope/e-shell/Products > tar -zxvvf ZSyncer-0.7.1-beta1.tgz
x ZSyncer/
x ZSyncer/www/
x ZSyncer/www/zsyncer_extra.gif
x ZSyncer/www/zsyncer_missing.gif
x ZSyncer/www/zsyncer_ok.gif
x ZSyncer/www/zsyncer_outdated.gif

[ ... ]

x ZSyncer/ZSyncer.py
x ZSyncer/__init__.py
x ZSyncer/changes.txt
x ZSyncer/utils.py
<wyrmslayer /usr/local/www/Zope/e-shell/Products > rm ZSyncer-0.7.1-beta1.tgz
<wyrmslayer /usr/local/www/Zope/e-shell/Products > ls ZSyncer/
Config.py               Extensions              ZSyncerTool.py          dtml                    version.txt
Config.pyc              LICENSE.txt             ZSyncerTool.pyc         help                    www
ConfigUtils.py          README.txt              __init__.py             skins                   zsyncer_icon.gif
ConfigUtils.pyc         TODO.txt                __init__.pyc            tests
ConnectionMgr.py        ZSyncer.py              credits.txt             utils.py
ConnectionMgr.pyc       ZSyncer.pyc             changes.txt             utils.pyc
<wyrmslayer /usr/local/www/Zope/e-shell/Products > cd ../bin
<wyrmslayer /usr/local/www/Zope/e-shell/bin > ./zopectl restart
. Unlinking stale socket /usr/local/www/Zope/e-shell/var/zopectlsock; sleep 1
. daemon process started, pid=4582
<wyrmslayer /usr/local/www/Zope/e-shell/bin >

Once the server has been restarted, go to your webapp folder in the ZMI, you will find a new tab (notice the sync tab on the right):

http://www.e-shell.org/img/samples/zsyncer_tab.png

All the sync process is managed within this tab, but first we have to create a ZSyncer? object inside our project folder:

http://www.e-shell.org/img/samples/zsyncer_select.png

You will have to give it an id and a title. The first one is important, because you will need it later to configure the sync, so give it a proper name like sync_with_remotehost (where remotehost is the hostname of your remote host) in your local host and sync_with_localhost (where localhost is the hostname of your local host) in your remote host. In this case, sync_with_Frey will be the ZSyncer? object created in wyrmslayer and sync_with_wyrmslayer will be the one created in Frey.

Once the Zsyncer object has been created, you can go to the sync tab, and you will see something like:

http://www.e-shell.org/img/samples/zsyncer_first_thumb.png

Ok, don't worry, the error appears because there isn't a valid configuration for the sync to be done. To create one, go to the Properties tab:

http://www.e-shell.org/img/samples/zsyncer_properties_thumb.png

The most important configuration parameter is Destinations, where you can set up so many remote servers as you want. Keep in mind that the remote server must have ZSyncer? installed, and a ZSyncer? Object must be created on such server too.

To add a remote server to the list, you must use an url like that:

http://user:pass@remotehost:port/path_to_ZSync_object

Mine looks like:

http://wu:********@Frey:8080/adea/sync_with_wyrmslayer

Being Frey my production server and wyrmslayer my development server (the one I'm currently working with).

You can leave the rest of the options with the default values, but perhaps you should check the Filter types? option. With it, you can limit which objects will be synced, instead of checking all the objects inside your webapp folder. In my case, for example, the webapp uses a PostgreSQL backend to store some data, so it uses a ZPsycopgDA? object to connect to such SQL backend. This object will be different on the different servers, so I do not need to keep it synced between them.

Remember that you have to repeat the process in both the production and the development servers (or each server you want to syncronize, remember that you can sync all the hosts you want), taking care of the ZSyncer? objects name. For example, the Destinations configuration parameter in my remote server looks like:

http://wu:********@wyrmslayer:8080/adea/sync_with_Frey

Once all the configs are ok, go to the sync tab again, it will connect to the remote server, checking the diferences between the webapp on both servers and the result will be something like:

http://www.e-shell.org/img/samples/zsyncer_check_thumb.png

Nice, isn't it?

As you can see in the screenshot, an intuitive layout is used to show the list of objects checked between the servers. In the list you can see the status of the object (with a nice set of icons), you can compare the modificacion dates between the different versions and you can even manage and view the object.

At the bottom of the list, you will find a legend explaining the meaning of the different icons:

http://www.e-shell.org/img/samples/zsyncer_legend.png

The green (ok) icon appears when the object is already in sync between the servers, the orange (out of date) icon appears when the object has been modified on the local host (the development server, wyrmslayer, in this case), the red (missing) icon appears when the object is in the local host, but not in the remote one (a new object on the webapp). Finally, the blue (extra) icon rarely appears in a case like this, it means that the object exist in the remote host (the production server, Frey, in this case), but it doesn't exist in the local host (perhaps an object not needed anymore?).

Keeping that in mind, if you look on top of the list, you will see some filters you can apply to change it:

http://www.e-shell.org/img/samples/zsyncer_opts.png

You can set if the sync process will check only for items on the current folder or if it must check items inside the subfolders too (take care that checking all items recursively will probably slow down the process) and you can set which kinds of objects you want to see. This is very useful, because you probably will be interested in see only which objects were changed and/or added in the development server version of the webapp, because those are the objects that will be upgraded in the production server.

http://www.e-shell.org/img/samples/zsyncer_filtered_thumb.png

So, in this case there are some new files (the ones marked with the red ?) and there are some files that had been changed. If you look just under the list, you will see five buttons. With those buttons we will do the sync. First select the objects you want to sync (or use the select all button to sync them all):

http://www.e-shell.org/img/samples/zsyncer_filtered_select.png

And then push the Put button. The objects will be sent to the remote server, a list of actions will appear progresively on the screen, followed by a Done button. Check that all the objects were sent and push Done. You will be back at the list, but this time all the objects must be marked as ok. Easy, isn't it.

From now on, every time you have finished a new feature, or fixed a bug, or whatever, you just have to go to the sync tab in your project folder, check the objects you want to sync, and push Put to upgrade the Production server version of your webapp in a clean-and-quick way.

Of course you can do more than just send files from the development server to the production one, you can Get files from the remote host, or check the diffs between different versions of an object, but i'll let you explore all this things by yourself.

Just a final NOTE! take care that ZSyncer? uses basic http auth in order to identify itself between the servers so, if you think someone could be sniffing traffic nearby (like syncing servers through a wireless link), **it could be a good idea to secure your Zope server using SSL.