MapProxy

New MapProxy 0.8.4 release

We are proud to announce the release of MapProxy 0.8.4. Read on for more information.

The latest release is available at: http://pypi.python.org/pypi/MapProxy

To update within you virtualenv:

$ pip install --no-deps --upgrade 'MapProxy<=0.8.99'

Note: The '<=0.8.99' prevents from installing 0.9 when it is released.

Updated documentation is available at: http://mapproxy.org/docs/0.8.4/

Release information

The 0.8.4 release contains minor bug fixes and improvements.

Some of the features of the new release are:

• Improved PNG performance with PIL fastpng branch.
• New concurrent_requests option to limit requests for each source WMS server.
• Extra newline at the end of all templates. Some deployment setups removed the last characters.

Improving the performance for PNG requests

MapProxy uses the Python Image Library (PIL) for everything image related: decoding, resizing, cropping, merging, transforming and encoding. Core parts of PIL are written in C and so is MapProxy able to keep up with pure C servers, and even trump them. Only the PNG encoding is significantly slower than other implementations.

There are two major variations of PNG files, often called PNG-24 and PNG-8. PNG-24 stores full color information for each pixel, while PNG-8 only supports a total of 256 different colors per image. The benefit of PNG-8 is that the files size reduces to 1/3. This process of reducing the possible 16.7 million colors of PNG-24 to a palette of only 256 colors is called color quantization.

We profiled the PNG encoder and found the bottleneck of the encoding process was in that color quantization. We've implemented a new algorithm, based on the octree algorithm, that offers better performance. Much better performance. The whole encoding process itself got about 10 times faster, increasing the MapProxy performance for PNG requests about 2-3 times, wich is now on a par with the JPEG format.

It even gets better: The new quantizer has also full transparency support. Prior to this, MapProxy could return images with transparency only in PNG-24 mode, but now transparency is also supported for PNG-8. For these images, the file size reduces to 1/4 of the PNG-24 size, while having the same performance boost.

How to use the new quantizer

The development of our modification is available at http://bitbucket.org/olt/pil-117-fastpng The modification is not yet included in the official PIL version, but the maintainer told me that it should be merged later this summer and that it should be available in the next PIL release.

But you can already use it anyway. There is a source package available and you can install and compile it with:

$ pip install http://bitbucket.org/olt/pil-117-fastpng/downloads/PIL-1.1.7-fastpng-a6.tar.gz

You need the latest MapProxy 0.8.4 release or the upcoming 0.9.0 to get support for the new quantizer.

By default, MapProxy will still generate PNG-24 for transparent images. To enable PNG-8 for these images, you have to add the following to your proxy.yaml:

image:
  paletted: true

Thats it. You should now have a MapProxy that is 2-3 times faster for PNG files, and with the reduced file size even faster for you users.

Have fun.

MapProxy Raster Image Benchmark

We did some benchmarks of MapProxy to show you how it compares to other WMS servers when it comes to serving raster images. This is a follow up to the WMS benchmark presented at the FOSS4G in Sydney and the additional tests done by Chris Tweedie.

We did not re-run all tests, but we have included MapServer as a comparison. You can use the results as a baseline to compare the MapProxy results with the numbers from the other benchmarks.

If you are not familiar with MapProxy, you should note that it is not a WMS server like MapServer or GeoServer, but a caching proxy for existing WMS servers. However, MapProxy does not cache every single response like an HTTP proxy such as Squid does. It builds up an internal tile cache and uses this cache to create new WMS responses. For raster images it basically does the same as a regular WMS server does: read raster data, resample the data to the right output resolution and deliver the result.

In this test we used the MapServer installation as the data source of the MapProxy.

Configuration

The test server is a virtualized XEN machine with 4 CPU cores and 7GB RAM running Ubuntu 9.10. The host machine has an Intel Xeon X3360 CPU with 4 cores at 2.83GHz. The test server was the only VM that was running during the benchmarks.

MapProxy

We used MapProxy 0.8.3 running with Python 2.6.4. It was deployed with gunicorn behind nginx as a reverse-proxy (no caching enabled). Gunicorn was configured to run 6 worker processes of MapProxy.

MapServer

We used MapServer 5.6.3. It was deployed as a FastCGI server behind Apache 2.2.12. We used 12 worker processes, which offered the best performance on this machine.

Test data

We used a Blue Marble Next Generation image as a data source for our tests. The image covers the whole World and is 21600 x 10800px. We used an uncompressed tiled TIFF with overviews (10800x5400, 5400x2700, 2700x1350, 1350x675). This should compare to the BigTiff dataset in the FOSS4G benchmark and to the Tiled (internal) from Chris Tweedies tests.

For MapProxy we used JPEG as the internal caching format and a tile size of 256x256 pixels.

Tests

We used the same BBOX extends and output dimensions that were used in the FOSS4G benchmarks. We took the 2000 random extends from bluemarble.csv and build a list of URLs for each tested server.

The image sizes vary between 256x256 and 1024x768 pixel. We tested JPEG as the output format with nearest neighbor resampling.

We tested different concurrency levels to see how the servers react under different loads. Each URL was requested once per test run and each test run was repeated three times. We used the best result of each test run.

Results

Ok, here are the results of our benchmark. At first the results for requests in EPSG:4326, the native reference system of the input data/cache:

MapServer can deliver up to 122 requests per second at 20 and 40 concurrent clients, MapProxy peaks at 40 concurrent clients with 157 requests per second, an increase of ~28%. Both servers can hold the performance with increasing concurrency.

And here are the results for requests in EPSG:900913, also known as the web mercator projection EPSG:3785. We used the same EPSG:4326 cache for MapProxy, so both servers need to reproject from EPSG:4326 to EPSG:900913.

MapServer can deliver up to 74 requests per second from 20 to 150 concurrent clients, MapProxy delivers 113 requests per second from 20 to 80 clients, an increase of ~52%.

Both servers showed very good results, and they exceeded our expectations.

Comparing with FOSS4G benchmarks

The MapServer results are much better than the results from the last FOSS4G benchmarks, 122 req/s to 28 req/s peak. There are two explanation for the difference. First, our test data set is only 900MB compared to the 16GB of images they used. We guess that the FOSS4G servers had some disk I/O during the tests that slowed the results down. Second, our test server is more recent and has more power.

Comparing with Chris' benchmarks

This one is more interesting. The CPUs are not that different, but his machine has two CPUs -- 8 cores. Still we got nearly three times as may requests per second. So either something was misconfigured on Chris's server or MapServer performs way better on Linux than on Windows.

Conclusion

You have to consider that both servers do the same job. As we already stated at the beginning, they just read raster data, resample the data to the right output resolution and deliver the image to the client. Therefore it is great to see that MapProxy can beat the oldtimer MapServer.

But is it worth to use MapProxy for a ~25-50% performance boost? Maybe not, but this was a simple test case. Serving raster images is the easiest job for MapServer, compared to rendering vector data from a shapefile or database. Rendering small scale maps with lots of data takes significant longer, render times of few seconds are not uncommon. The performance of MapProxy, on the other hand, is independent of the data. So, while the benefits of MapProxy seam small in this test, they can become huge when you start to serve larger raster or vector datasets.

Come join our mailing list if you want to discuss the results, or leave a comment below.