Twisted Logo

in python, software, twisted

Making and deploying a Twisted project as a service under Windows

What is the goal?

The goal is to implement a program in Python+Twisted (using PB for network access) under Windows XP or 2000+, that can be run before a user logs on, so it has to be a windows service, launched automatically, at boot. Another goal is to show some developement patterns in Twisted. You will find a lot of ‘theoretical’ patterns about how to make singletons/borgs, proxy, and stuff, but I never found patterns about ‘Twisted code’, except for the wonderful ‘finger tutorial’ by the squishy moshez. This tutorial can be split in two parts: The first one is about writing a good skeleton for your Twisted development. The second part is about making a Windows service.

Note that these instructions (the service building part) will not work under Windows 98, 95 or older.

Required packages

On the Windows machine, we have two categories of requirements:

  • The required stuff for generating the files: To do so, we need: ActiveState Python 2.3+ (because it includes the latest version of the modules pywin32), Twisted 1.3+, and py2exe 0.5+, and eventually gvim for editing the files.
  • The required stuff when we have generated all the required files: nothing, as we’ll have a compiled program which includes all the required DLLs.

Installation of the required packages

Nothing special here, just click, except that I had to add the path to the scripts (twistd) in Python2.3 (installation path to Python and path to scripts directory).

Writing the program

In our program, we don’t need anything specific that will require the use of the specially made reactor for win32, in fact, there’s not really any reason to use it anyway. There’s just the case if you want to use the utils.getProcessValue() method of twisted which doen’t work if you use the default select recator. That’s not really a problem, since you can use the win32pipe.popen() from the pywin32 modules which works perfectly in Twisted (at least for me).

Here, our program is a simple one that just listens on port 1234 using PB, making the method getFoo() available to the outside. This method just returns the current time.
Simple, Fast and short code

Here is the simplest code, without using tricks to shorten the code making it unreadable: Myprog1.py

How to run this code: Simply type python Myprog1.py.

The method requestAvatar of the class SimpleRealm is called whenever a user (a pb client here) has been authenticated. This method associates the user to a class (here Presence) which contains the allowed remotely callable methods (identified by perspective_*). In this example, there is just one method that can be called: getFoo. A PB factory is created to respond to external requests, and map successful requests to the SimpleRealm class.

To test this code, you can write a simple PB client that will ask for the getFoo method and get the result. I’ve done it for you in the file miniclient.py:

If you are not familiar with the deferred mechanism, you can read the corresponding HOWTO on http://twistedmatrix.com/projects/core/documentation/howto/defer.html.

Cleaner, more open, but more verbose code

For a cleaner, more ‘Production grade’ source, you can look at this one: Myprog2.py

How to run this code: Simply type python Myprog2.py.

The difference from the previous Myprog1.py program is that a new class Agent is created, a subclass of service.Service. There are multiple reasons for doing this:

  • It allows us to have a “central class” where you can store your common methods to all your eventual different Presences and other methods. You could store them in SimpleRealm (and you’d be right if your program is really small) but I don’t think that’s really clean. The way I see it, the class related to realms is just supposed to contain classes related to user management.
  • If you want to use twistd, .tac and other fun Twisted stuff, you have to get an application defined. The first child of this application would be an instance of the Agent class.

This code is much cleaner, but longer, but allows to continue developping without barriers like communication between services, etc. This version is launched just by running python and the .py file.

Code shaped for usage with twistd

In Twisted you have the possibility to launch a .py file with a wrapper, named twistd (note the absence of ‘e’). But to do this, you need to initialize things differently from the method shown in Myprog2.py. Here’s the code: Myprog3.py

How to run this code: use twistd -noy Myprog3.py.

Here, what happens is that twistd looks for an object named application and uses it. So you ABSOLUTELY need to have that object named application. There are some modifications compared to Myprog2.py, where SimpleRealm is more verbose and allows to create different presences.

Dispatching code in different files/modules

The code is a little longer, making it more difficult to read. So I made another version, where code is split in two files: Myprog4.py and Myprog4Presences.py

How to run this code: use twistd -noy Myprog4.py.

Besides splitting the classes in different files, I added an example of perspective (perspective_getFromCommon) that goes to the “main application” (in fact the root of the tree where you have the app, then the simulation realm, and then the presences) to execute a ‘common’ method. Note: This code uses the ex-manhole stuff (now ShellFactory) to allow connection to the program while running. I’ll write a small tutorial on this later.

Using .tac file to finish the skeleton

Finally, we’re at the last step of achieving our ‘perfect pattern program’ in Twisted. The last thing to do, is to create a .tac file. Why? Well, it’s not useful, but it’s a requirement for building a service under Windows. A .tac is really simple: it’s just the ‘main’ part of the program put in a separate file with an extension ‘.tac’: Myprog5.py, Myprog5Presences.py and Myprog5.tac

How to run this code: use twistd -noy Myprog5.tac.

Final note and update

As you surely know, Twisted is a fast moving target, it seems it is now the usage not to use internet.TCP/UDP* anymore, but the method strports.service(), which will surely allows in the future to pass the information on the twistd command-line.

As an example, now you just need to replace the following code:

With:

Making a service of the program

(This section is now specific to Windows).

How to create a service, having a python program? The only way I know is to use the py2exe program whose initial goal is to find all the dependencies (modules and libraries), make an archive of it, and create an executable of the main program. The goal is to be able to easily distribute some python program without having requirements like ‘python must be installed, wxPython and wax modules have to be present on version >x.y’. Another advantage of py2exe is to allow the developer to choose how your program is compiled. It can be of three formats: console, window, com_server or service. For testing purposes, it is best to use ‘console’ mode, which, when your program will be launched, will open a Windows shell and run your program inside, making it easy to see any exception or other error.

To use py2exe, we’ll have to create a setup.py file because py2exe uses the nice distutils module (which should always be installed with python by default on ANY distribution).

We’ll first try to compile our program as a console app. Normally, for a normal python program the setup.py file would be as simple as running python setup.py build:

When this works correctly, you can continue (the hard part) to transform your program into a Windows service. Now that your program runs in a console, we’ll make it a service. What are the advantages of making of your program a service? Simple, you can launch your service before anyone logs on, at the machine boot, that means you’re not attached to the presence of a user logged on. That allows you to secure a little more you application by setting the owner/runner of the service a least privileged user. Another reason is that you can easily start/stop/restart your service graphically by clicking in the Windows Service Manager.

For a normal python program that’s not really a problem, here’s the corresponding setup.py file:

Here, you’ll need to have an attribute _svc_name_ in a module named MyService.

OK. That’s easy, you say, and you’re right. The only problem is that this doesn’t work if you’re using Twisted. The main reason is that that class containing the _svc_name_ has to be a loop, where you have a start, a stop, etc. That means you’ll be stuck running this loop, and not the reactor :-(

Moreover, we want to be able to run our program which is in a .tac, and with twistd. So we’ll have to use a supplementary module, written by Moonfallen, available in his svn sandbox (svn co svn://svn.twistedmatrix.com/svn/Twisted/sandbox/moonfallen). You can view it using the web interface on http://svn.twistedmatrix.com/cvs/sandbox/moonfallen/ . You’ll get the following files:

Install them in the directory where you put your setup.py file. and unzip modulegraph.zip in your site-packages directory (follow the instructions in the README).
Bug in Py2exe 0.5.4

If you just installed Py2exe without patching it, you will certainly end up with an error when you’ll try to run your program (a can’t find module linecache or os). As written in moonfallen’s README, you have to patch the boot_service.py file which should be in C:Python24site-packagespy2exe:

How to write the setup.py file

Modify the setup.py to use ntsvc:

And then run python setup.py twistedservice. You should have an output that looks like this:

The missing libraries at the end of the generation is not a problem if you don’t use them (they’re mostly crypto related).

What you get

What python setup.py twistedservice does is:

* Create a directory named dist (and build, which is less important)
* Analyzes all the dependencies of your python code
* Copy all the dependent modules and dependent libraries to that dist directory
* Generate a library.zip with all the libraries.
* In our case (because we chose to use ‘service’ in the setup.py file), we got an executable with the name of our .tac with ctl appended to it.

Here’s an example of the content of the dist directory:

For example, if our .tac file was agent.tac, we would end up with an executable named agentctl.exe. if we run it, we see that it’s a “control program” to add/remove our newly made service:

Connecting to the Service Control Manager

If you run agentctl.exe -install, and then look in the Windows Service GUI, you should see a new line:
images/service.png

You’ll just have to right-click on it and click on Start Service.

Add-on: how to make an installation file

OK, now you have a directory that you can distribute, without having to require Python, Twisted and other dependencies. But it’s a directory. You can zip it, but it’s not very efficient, especially if you have to be able to remove it later, or deploy it on several machines.

What I recommend is to use the new installation format, used by the official Python package too, that is, MSI.

Generating an MSI package

Generating an MSI file is really easy, especially if you’re using Windows XP, as the kit is already included, otherwise you’ll have to download it from Microsoft’s website.

To summarize: to generate an MSI you’ll have to create an xml file where you have quite a lot of variables (program path, name of program, files to copy, etc.). This is quite long, and, without a GUI that does it for you, it’s nearly impossible. That’s why I recommend you Installer2Go. It’s a really nice freeware (there’s a registered version if you want support) that allows you to build your .msi all by just clicking:
images/i2g2.png

You can even define the automatic registration and the removal of you service:
images/i2g1.png

That’s all for now. It’s the first release of this document, so there may be some typo errors. I checked all the Myprog*.py, they all work on my machine (Python2.3 or Python2.4, Twisted 1.3 or Twisted 2.0.1).

References

  • http://starship.python.net/crew/theller/moin.cgi/Py2Exe
  • http://svn.twistedmatrix.com/cvs/sandbox/moonfallen/
  • http://twistedmatrix.com/~moonfallen/
  • http://svn.twistedmatrix.com/cvs/sandbox/moonfallen/README.txt?view=auto
  • http://www.installsite.org/pages/en/msi/authoring.htm
  • http://www.dev4pc.com/installer2go.html
  • http://www.qwerty-msi.com/
  • http://www.aksdb.org/downloads.php
  • http://msi2xml.sourceforge.net/
  • http://desktopengineer.com/index.php?topic=0070MSI
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Write a Comment

Comment