Tirex/Overview
The Tirex tile rendering system renders map tiles on the fly and caches them. It is very similar (and mostly backwards compatible) to the mod_tile/renderd system, but improves on it in many respects.
Rendering backends
Tirex can work with several different rendering backends at the same time. Normally Mapnik is used for this, but a test backend (to test the system setup) and a WMS backend is also provided.
Maps
Tirex can handle several maps. Maps can have different styles and potentially use different data. In the most common case different maps are just different map.xml style files for the Mapnik renderer.
Tiles
As with many other map systems, the map of the world uses a Mercator projection and is available in several zoom levels and divided into tiles. Tiles are always square and 256x256 pixels. At zoom level 0 the map of the whole world fits into one tile, at zoom level 1 there are four (two by two) tiles for the whole world. At zoom level 2 there are 16 tiles and so on. The highest zoom level depends on the detail you want. For OSM its normally about 17 to 19.
Tiles are numbered x=0 (-180 degrees) to x=2^zoom-1 (+180 degrees) and from y=0 (about 85 degrees) to y=2^zoom-1 (about -85 degrees).
Metatiles
The tile size (256x256) is optimized for fast transfer on the web, it is not necessarily the best size for handling tiles on the tile server. Rendering many small tiles takes more time than rendering fewer bigger tiles and storing many small tiles in single files is inefficient because of the block size used by filesystems. Also a directory with many files in it will be slower than with fewer files.
This is solved by aggregating several tiles into a metatile. Typically 8x8 tiles make up one metatile, but this is configurable (although sizes other than 8x8 are not well tested) in Tirex. This is the same size as used in mod_tile/renderd systems. The tile server internally always "thinks" in metatiles. Access requests for tiles are converted into requests for metatiles. Metatiles are rendered and stored.
Metatiles are numbered just like tiles, a metatile number is always the same as the number of the top-left tile, in other words to get the metatile number you have to round the tile coordinates down to the neirest multiple of the metatile size (8). Tile (x=17 y=4) is metatile (x=16 y=0).
In the Tirex system metatiles are represented by Tirex::Metatile objects. In addition to the x- and y-coordinates and the zoom level, a metatile also needs the map name.
Jobs
If a metatile should be rendered, a job must be created for it. The job contains at least the information about the metatile and the priority. It can also contain additional information such as the expire time, the source of the request and the time it took to render the job (once it is rendered).
In the Tirex system jobs are represented by Tirex::Job objects.
The master will keep track on where a job came from so that it can notify the source when the job is done.
The Queue
The master keeps a prioritized queue of jobs. Every job that comes in will be placed on the queue. When there are free renderers, the master will take the first job from the queue and sends it to the renderer backend.
There is only one queue for all jobs, but it is prioritized. Priorities are positive integers, 1 is the highest priority. New jobs are added before the first job in the queue with a lower priority. Jobs are always taken from the front of the queue. So jobs will always be worked on based on their priority and age.
A job can have an expire time. If it comes up to be rendered and the expire time has passed, the job will not be rendered. This basically allows you to say: I have a metatile I want rendered because I might need it in the next few minutes. But if its not rendered in, say, 10 minutes, you needn't bother rendering it.
When a new job comes in for a metatile that already has a job on the queue, those two jobs will be merged. The old job will be taken from the queue and a new job will be added. The new job has the higher priority of both jobs. The expire time of the new job will be the larger of both times. No expire time is the same as "infinite expire time", so if at least one of the jobs has no expire time, the new job will have no expire time. Both sources of the two jobs will be notified when the job is rendered.
The queue is implemented in a way that its doesn't matter how many jobs there are in the queue. If you want to stick 1000 or 10000 jobs in the queue, thats ok.
It is your job as administrator of the system to decide which priorites to use for which kind of requests. Live requests coming in from a user should probably get a higher priority than batch requests to re-render old tiles. The Tirex system gives you the mechanisms needed, you have to decide which jobs get priority, how long they should stay on the queue etc.
Buckets
Tirex allows an infinite number of priorities. To make configuration and handling easier, these priorities can be divided up into several buckets. Each bucket has a name and represents all priorities in a certain range. You define the name and range. Configuration and some other operations will use those priority classes instead of the priorities itself.
A typical setup will have a bucket for live requests from the web (lets call it 'live') that works on priorities 1 to, say, 9. And then one or more buckets for background requests with lower priorities.
The Master
The master is the heart of the Tirex system. Its job is to work throught the queue in order and to dispatch jobs to be rendered when its their turn. The manager takes the configuration and the current system load into account when deciding which and how many tiles can be rendered.
Messages
The Tirex system consists of several processes that work together. Requests for the rendering of metatiles and other messages are passed between those processes through UDP or UNIX domain datagram sockets. Datagram sockets are used to make handling of many data sources easier in a non-blocking environment.
Messages always have the same, similar format: Fields are written as "key=value" pairs (no spaces allowed), one per line. A linefeed (LF, "\n") is used as a line ending, the software ignores an additial carriage return (CR, "\r") before the linefeed. Each message must have a "type" (for instance "type=metatile_request"). Requests will normally be answered by a reply with the same type and added "result" field. The result can either be positive ("result=ok") or negative ("result=error"). More descriptive error messages are also allowed, they always begin with "error_" ("type=error_unknown_message"). An additional error message for human consumption can be added in the "errmsg" field.
Request handling
The following simplified diagram shows how a tile request from a web browser is handled by Tirex:
- The web browser ask the webserver (in this case Apache with the mod_tile module) for the tile.
- mod_tile checks the disk for the tile. If it is available, it can be delivered to the browser immediately (9). If it is not available we go on...
- Send a request to tirex-master for the tile. The master will put the request in the queue.
- Once its the turn of this tile, the master will send the request on to the rendering backend, typically the one using the Mapnik renderer tirex-renderd-mapnik. There are other backends available, too.
- The rendering backend generates the tile and stores it on disk.
- It then sends a reply back to the master to tell it that the tile is done.
- The master sends this reply on to the original source of the request.
- mod_tile now gets the tile image from disk...
- ...and sends it back to the browser.