nabeel shahzad

Shell scripts no more!

with 14 comments

This weekend, I started my server migration, over to Slicehost. It went well, now I’m running on a lean ‘n mean nginx install. As I was moving my Subversion repositories, I was dreading having to move all my shell scripts, which I used to build and deploy some of my applications (outlined in this post). I was thinking there had to be a better way, after all, Ruby has Capistrano, and though it can be used with PHP, I didn’t want to have to install Ruby, etc etc. After some searching (not much), I found Phing, which looked like exactly what I needed. Sweet!

Phing takes an XML file, which you can define all the transformations and instructions. It has sets called “targets”, which are different stages of the build and transformation. They have a basic example here, but I’ll go through my phpVMS build.xml, since I do a number of things they don’t show in the examples, like checking out from Subversion, some variable replacements, and building tars for different “stages”. Having targets was great, since you can select which targets to run, so now I can combine a “beta” and “release” builder, instead of having separate files for it (as I did with the shell scripts). My goals are the same as in the SSH scripts:

  • Automatically build beta, full, and update versions from SVN
  • Update revision numbers within several files using tokens or regex
  • Copy it to the test site
  • Automatically generate PHPDoc API documentation

Phing is perfect for this. I had sets of painful-to-maintain and update SSH scripts to do this same thing. And it has a simple PEAR installer. I installed it using:

Java
1
2
sudo pear channel-discover pear.phing.info
sudo pear install phing/phing

]

And that was it! Installed. They do have instructions for a I now had “phing” available on the command line:

Java
1
2
phing -version
Phing version 2.3.3

]

Now we can get started, building our XML file. The file starts with:

Java
1
2
3
4
5
<xml version="1.0">
<project basedir="." default="beta" name="phpvms">
</project>
</xml>

That’s our basic container; as you can see I set the project name, the default “target” or stage to run, and the base directory.  Everything we do/add will go inside the project tags.

Now, I’m going to add some common “properties”, or variables, that will be used throughout the script, to make it easier, and not have to type out paths over and over:

Java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<!-- Build properties -->
<property name="project.wcdir" value="/path/to/phpvms.wc" />
<!-- This is the path to where a clean export can be done of SVN -->
<property name="project.exportdir" value="/path/to/phpvms.export" />
<!-- This is the path where the build binaries can go -->
<property name="dest.dir" value="/path/to/public/files/" />
<!-- This is the path where the "test build" goes (phpvms.net/test) -->
<property name="test.dir" value="/path/to/public/test/" />
<!-- This is where the docs generated by phpdoc will go -->
<property name="php.doc" value="/path/to/public/docs/api/" />
<!-- URL to our SVN repository -->
<property name="repository.url" value="svn://phpvms.net/phpvms/trunk" />
<!-- Username and password, blank since the above can be accessed anon -->
<property name="repository.user" value="" />
<property name="repository.pass" value="" />

Use absolute paths if you can, otherwise, it’s relative to the basedir property defined in the project tag.

The reason for having a working copy and an export directory for SVN is this – the checkout directory will be used to determine the latest revision, since the svnlastrevision command requires the path to a working copy. The export directory is to hold a clean export; without all the subversion specific data that’s included in a checkout. We’ll package that up, and use that to copy to a test site.

Next, I define the targets, or stages, I’m going to use: prepare, build, subs, phpdoc, beta, and release.  They are setup this way in roughly the order they will run. The order doesn’t matter, since we determine the order of the targets by using the “depends” property. I’ll explain each one in detail:

Java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<target name="build" depends="prepare">
</target>
<!-- This is where we will do our variable substitutions
  It depends on the target above, primarily in our case,
  replace all version numbers -->
<target name="subs" depends="build">
</target>
<!-- This is the PHP doc target, it's its own target
  since we will call it from beta and release -->
<target name="phpdoc">
</target>
<!-- In this target, we will create a "beta" release.
  It depends on the subs build. We will call this
  target, or the release target specifically from the
  command line when we run phing -->
<target name="beta" depends="subs">
</target>
<!-- In this target, we create a "release", and it depends on
  the "subs" target above. We'll also specifically call this
  from the command line -->
<target name="release" depends="subs">
</target>

So now that we have an outline, we can go through each target, doing what we need to do, first our “prepare” stage:

Java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
<target name="prepare">
    <!-- First we are going to delete the export directory
      So we have a clean copy that we export from SVN.
      Then we re-create it using the mkdir command -->
    <delete dir="${project.exportdir}" failonerror="true" verbose="false" includeemptydirs="true" />
    <mkdir dir="${project.exportdir}" />
    <!-- Now we create the working-copy directory,
      where the checkout will be -->
    <mkdir dir="${project.wcdir}" />
</target>

We’re using our properties which we defined, as ${propertyname}. Don’t need to do any escaping, etc, just use ‘em right in the strings.

Next is our build target stage, which will do the Subversion operations of getting the latest revision.

Java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<target name="build" depends="prepare">
    <!-- We are checking it out to the checkout directory -->
    <svncheckout todir="${project.wcdir}" repositoryurl="${repository.url}" nocache="true" force="true" password="${repository.pass}" username="${repository.user}" svnpath="/usr/bin/svn" />
    <!-- We're now going to get the last revision
      from the above checkout. The last revision
      number will be stored in the propety called
      svn.lastrevision, which we can access by
      ${svn.lastrevision}. No more parsing results
      from the command line... wahoo! -->
    <svnlastrevision svnpath="/usr/bin/svn" propertyname="svn.lastrevision" workingcopy="${project.wcdir}" />
    <!-- Export a clean copy which we can package up -->
    <svnexport todir="${project.exportdir}" repositoryurl="${repository.url}" nocache="true" force="true" password="${repository.pass}" username="${repository.user}" svnpath="/usr/bin/svn" />
</target>

It’s great how Subversion operations can easily be done. Next target is subs, to replace any tokens and variables we define in the files.

Java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
<target name="subs" depends="build">
<!-- Reflexive allows work on a set of files, and
  a "reflexive" block is required for a filterchain-->
<reflexive>
<!-- Here we define the files that we're going to
   work in using a filterchain -->
<fileset dir="${project.exportdir}">
    <include name="changelog.htm" />
    <include name="install/install.php" />
    <include name="install/update.php" />
    <include name="install/install.sql" />
</fileset>
<!-- Setup a filterchain now, the above files
   pass through these filters -->
<filterchain>
    <!-- Instead of doing a regular expression match,
      Phing supports token replacement. I use the
      ##REVISION## token wherever I want it to be -->
    <replacetokens endtoken="##" begintoken="##">
   <!-- Replace the REVISION token with our svn.lastrevision property -->
   <token value="${svn.lastrevision}" key="REVISION" />
    </replacetokens>
</filterchain>
</reflexive>
</target>

Next is our phpDoc function, which is sort of standalone, in the fact that there are no depends for it.

Java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<target name="phpdoc">
<!-- Setup the basic options -->
<phpdoc title="API Documentation" output="HTML:Smarty:PHP" sourcecode="no" destdir="${php.doc}">
<!-- These are the files that are going to be
   included as part of the documentation -->
<fileset dir="${project.exportdir}">
    <include name="core/classes/*.php" />
    <include name="core/common/*.php" />
    <include name="core/modules/**/.php" />
</fileset>
<!-- This lists the files which are part of the
   official documentation -->
<projdocfileset dir="${project.exportdir}">
    <include name="changelog.htm" />
</projdocfileset>
</phpdoc>
</target>

Now that we’ve got the basics down, the next two targets are “beta” and “release”. Each of these handles a different set of tasks, though they are basically the same. First for beta:

Java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
<target name="beta" depends="subs">
    <echo msg="Creating archive..." />
    <!-- Delete the files first, since they are not updated,
      The tar function will just append onto them -->
    <delete file="${dest.dir}/phpvms.beta.tar.gz" />
    <delete file="${dest.dir}/phpvms.beta.zip" />
    <!-- Create the tar file, specifying the files to include -->
    <tar compression="gzip" destfile="${dest.dir}/phpvms.beta.tar.gz">
   <fileset dir="${project.exportdir}">
        <include name="*" />
   </fileset>
    </tar>
    <!-- I was having trouble with the built-in ZIP functionality
      So I resorted to using the command line. Either way, this
      shows off the command line functionality that Phing has -->
    <exec escape="false" command="cd ${project.exportdir}; zip -D -r ${dest.dir}phpvms.beta.zip ." />
    <echo msg="Copying to test site" />
    <!-- Copy it to the test site, so there's an updated
      copy available there to debug-->
    <copy todir="${test.dir}" overwrite="true">
   <fileset dir="${project.exportdir}">
        <include name="*" />
   </fileset>
    </copy>
    <!-- Call the phpDoc target that we created -->
    <echo msg="Creating phpDoc" />
    <phingcall target="phpdoc" />
    <echo msg="Files copied and compressed in build directory OK!" />
</target>

And now for the release target. Release a bit different than beta – it creates two different archives – one full version, and one update. The difference essentially is that the update copy has the local configuration file removed, where database settings are stored, and other settings which are kept locally.

Java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
<target name="release" depends="subs">
    <echo msg="Creating release builds..." />
    <!-- Delete all the files, start fresh -->
    <delete file="${dest.dir}/phpvms.full.tar.gz" />
    <delete file="${dest.dir}/phpvms.full.zip" />
    <delete file="${dest.dir}/phpvms.update.tar.gz" />
    <delete file="${dest.dir}/phpvms.update.zip" />
    <!-- Create the tar file -->
    <tar compression="gzip" destfile="${dest.dir}/phpvms.full.tar.gz">
   <fileset dir="${project.exportdir}">
        <include name="*" />
   </fileset>
    </tar>
    <!-- Create the zip file -->
    <exec escape="false" command="cd ${project.exportdir}; zip -D -r ${dest.dir}phpvms.full.zip ." />
    <!-- Now create the update build, which basically deleting the
                local.config.php file -->
    <echo msg="Full release created, creating update" />
    <!-- Delete the local.config.php file, since that's not distributed
      with an update-->
    <delete file="${project.exportdir}/core/local.config.php" />
    <!-- Create the tar file -->
    <tar compression="gzip" destfile="${dest.dir}/phpvms.update.tar.gz">
   <fileset dir="${project.exportdir}">
        <include name="*" />
   </fileset>
    </tar>
    <!-- Create the zip file, using command line instead
       I kept getting out of memory errors using the built-in
       ZIP functionality. -->
    <exec escape="false" command="cd ${project.exportdir}; zip -D -r ${dest.dir}phpvms.update.zip ." />
    <!-- Call the phpDoc target -->
    <echo msg="Creating phpDoc" />
    <phingcall target="phpdoc" />
    <echo msg="Files copied and files moved, done!" />
</target>

And that’s it! Now we can run it using:

Java
1
2
phing -buildfile /path/to/build.xml beta
phing -buildfile /path/to/build.xml release

]

Using the appropriate one we want to use. In my SVN post-commit file, I run the beta line, in order to have that build every time.

I attached a fully doc’d version here (same as the above, just in one file). I hope it helps!

Be Sociable, Share!

Written by Nabeel

February 16th, 2009 at 5:49 pm

Posted in php,phpVMS,Projects

Tagged with , ,

«
»
  • http://www.toosweettobesour.com/ Daniel Cousineau

    Setting up build properties can be a lot easier.

    1. You can use self closing tags on the properties IIRC. e.g.
    2. Even better, you can have a build.properties plain text file of the format:

    # Comment
    variable.name=value

    And then link to it via a

  • http://www.toosweettobesour.com/ Daniel Cousineau

    Well it appears my last comment had the tags stripped out.

    You can link to the properties file via a <property file=”./build.properties” />

  • http://om4.com.au James Collins

    We currently use subversion with a collection of bash scripts for deployments, however after reading your post I’m interested in possibly using Phing instead.

    Have you had any experience with building the “beta” from trunk and the “release” from a given branch or tag in the repository? Is this possible to do with Phing?

  • Nabeel

    @James Collins
    Yes, you could do that easily. Just move the code fo SVN checkout into the “beta” stage instead, and point it to trunk. Then another checkout in the “release” stage pointing to a branch or tag. I’ve been planning to post another example, where in release, I do a checkout from trunk, and then create a tag from it using the “svnupdate” task (by checking out the tag, copying the files, then updating, a bit round about).

    @Daniel Cousineau
    Thanks for that info, I’ll check it out. I’m digging deeper into Phing now, I’m getting all sorts of ideas

  • Federico

    Make sure to visit Raphael’s blog, he’s the Phing guru :)

    http://raphaelstolt.blogspot.com/search/label/Phing

  • http://www.plankdesign.com meeech

    Hey
    Its undocumented, but you can point svnlastrevision at a repository, and it works.

  • http://www.plankdesign.com meeech

    (tag got stripped out)
    [ svnlastrevision repositoryurl="${svn.repourl}" username="${svn.username}" password="${svn.password}" propertyname="svn.lastrevision" / ]

  • Nabeel

    @meeech
    Really? I’ll have to try that out. Hurray for undocumented features :D

  • http://www.fbollon.net fred

    You could give a try to fredistrano -> http://code.google.com/p/fredistrano/
    it’s a deployment tool for web application.
    Fredistrano export your sources from subversion and synchronizes them with your production server. It is possible to execute several tasks during the deployment process as renaming files, modifying permissions and executing customs shell scripts.

  • Pingback: Weekly Links #42 | GrantPalin.com

  • http://www.thedeveloperday.com Žilvinas

    It seems to me your svn server is on the same server as your application. That is quite odd to be honest ;) If not then i don’t understand how you execute phing from your post-commit file (unless you connect through ssh there).

    We have many websites and it would be a pain to log in to the server each time you want to run phing. Maybe it would be posssible to execute phing through ssh with custom scripts?

  • Nabeel

    @Žilvinas
    Yes, it’s on the same server in this instance, but not all my sites are on the same server, which I use Phing for all of them now. The post-commit file just calls Phing to build (phing -f /path/to/file). To deploy on other servers, what I do is use the ExecTask to call scp, to copy the files to the remote server, after I’ve done the replacements, etc. Basically the same as the CopyTask.

  • Pingback: Phing automatise les opérations répétitives des développements PHP | Pims Labs

  • http://lebenplusplus.de/ Gabriel

    Hi,

    When installing phing with pear you should use the -a flag – it installs useful optional dependencies:

    sudo pear install -a phing/phing