morungos

Most of the time CPAN, and its smaller friend, App::cpanminus, are great for setting up all your Perl modules. I've just managed to complete a new Perl build, more or less equivalent to Strawberry Perl, and it only took a couple of days. Most of this was very easy, thanks to MinGW64 becoming a great replacement for MinGW. However, a couple of things were bad, and are worth mentioning.

Before I do that, it's worth saying that I have been through this process before with MinGW - and believe me, it used to be much worse. A few modules which I had expected to be a total pain (yes, I'm looking at you DBD::mysql) were trivial and automatic to install. Even XML::LibXML and XML::LibXSLT, which on occasion had been show-stoppers, were pretty straightforward.

The biggest problems were with OpenSSL. Here, I made the slightly stupid move (only in retrospect) of using dynamic libraries rather than static ones. Since I have around six modules that link against OpenSSL, this seemed the right move, as I'd only have one copy of the code in memory. However, the library names for the dynamic library version are different (i.e., libcrypto.dll.a rather than libeay32.a, and libssl.dll.a rather than libssl32.a), and all the CPAN modules were wired to use the static names, so things went bad spectacularly. The modules included at least Crypt::OpenSSL::RSA. Crypt::OpenSSL::X509, Crypt::OpenSSL::Random, Net::SSLeay, IO::Socket::SSL, and Crypt::SSLeay. With this number of modules sharing code, it seems reasonable to use dynamic libraries.

In practice, virtually all of these failed to find the Windows OpenSSL libraries I wanted. In fact, they all seemed to have different code for finding the libraries. Some used Module::Install, some had hard-wired library names in ExtUtils::MakeMaker. This seemed a lot of duplicated effort, and also a huge risk. (I even saw in the CPAN Testers wiki that sometimes people use one of two different SSL modules, in case one doesn't install right - this sounds to me like this problem is not uncommon, and that a fair few people are hitting OpenSSL module issues.)

Unlike UNIX and its related systems, where libraries are usually in a few sensible places, in Windows, they can be virtually anywhere. Many installers looked for C:/openssl/bin/openssl.exe, and ran it, to find out where stuff was. Or some looked in the PATH environment variable to find openssl.exe to run. Others assumed the library path would be right, which at least allowed me to override on the command line in a manual install. However, in most cases, the installer would (a) pick the wrong path, and (b) have no mechanism to allow me to specify the right one, except by manually hacking Makefile.PL before I do a build.

So this got me thinking. Is this a CPAN issue? Should I be able to instruct ExtUtils::MakeMaker where stuff actually is? Should I be able to do this even overriding an installer? Or would I be able to go round all the OpenSSL module authors and offer a module which would find OpenSSL properly once. This last option has a lot of appeal, and would probably be the right strategy, if achievable. I kind of feel like I've been bitten by Tim Toady on this one.

And before anybody comments that I should just not use Windows: people, that is not an option. This is not my choice. We have a large-ish user base that requires Windows. One time, it took us four months to get permission to unblock a TCP port. Another client demanded a detailed justification for all use of open source software (yes, really!). I would switch to *BSD or Linux tomorrow if I could.

I'm thinking about Perl, and how people can possibly believe the community is anything less than vibrant. As some may know, I used to be a Common Lisp developer, and Common Lisp has a lot in common with Perl. I haven't done too much with Lisp for the past ten years, but when I did work in it (between 1985 and 2000, more or less) all the time there were questions over its survival. Sorry, everyone, Common Lisp is still here.

There is a reason why Perl has survived so well, and why it will continue to do so. The reason is subtle, and one of the less apparent aspects of Perl, and one that is sometimes criticized, its very malleable syntax.

Common Lisp has macros, and a relatively small core of 'special forms' which were all the compiler needed to know about. Virtually all the visible syntax of Lisp was macros, written to expand into these special forms. For this reason, very significant aspects of the language could be added without changing the core compiler in any way.

Initial versions of the Common Lisp Object System, or CLOS, were written in exactly this way. In the early days, objects were represented as fancy closures - not great performance, but a very good way of encouraging community participation with CLOS, and also of smoothing off any rough edges. Over time, this became more integrated. The Common Lisp exception system also went through a similar transition.

Now consider Perl. Perl has accommodated similar inclusions, and subtle changes in the parsing since Perl 5.8 have made this easier.

Let's face it, Perl's exception handling is on the clumsy side - but the CPAN module Try::Tiny managed to turn it into a much clearer code form, without requiring the core language system to be changed. In a language with a more conventional syntax system, control structures cannot generally be added without core system support. Try::Tiny looks like a control structure, but it's actually a function that does some sleight of hand with closures and functions - exactly how the prototypes of Common Lisp exception system retrofitted similar structures.

Object systems are similar. Perl's initially simple object system has blossomed into one of the smoothest and most flexible object systems (Moose) in any language in mainstream use. Moose also supports the "syntactic sugar" which makes it seem like these features were there from the beginning.

This characteristic means Perl can evolve in ways that are hard to predict. Like Lisp, Perl is a survivor, able to accommodate almost any idea adopted by other languages - and like Lisp, when Perl accommodates a new feature, it is very likely to improve on the original source. CLOS is almost unrecognizable as a descendant of Smalltalk, yet that was the source for many of its concepts. Perl's strategy of "there's more than one way to do it" demanded this syntactic flexibility, yet its by-product is language extensibility.

By contrast, most languages have very strict grammars. Java's grammar is especially well-defined, and in addition, by eliminating any form of preprocessing, it puts the entire burden of language change onto the compiler. PHP, Ruby, and Python all share this syntactic inflexibility, although to a lesser extent than Java. The problem is: this places the burden of language extension on core developers rather than module developers - and not just the burden, the control too. In effect, the direction the language takes will have to be driven by core developers rather than a wider community. Paradoxically, the language may be more likely to fork with a strict grammar, rather than less.

Perl didn't go as far as Lisp with its macro system (well, actually it did, with filters, but these are a very powerful tool, and best used very cautiously). But the principle of a malleable and accommodating syntax means Perl will continue to evolve, and to surprise as it does evolve. Perl is not only here, but it has a surprisingly bright future.

My first ever real job was to develop code which handled dates and times. today, more than 25 years later -- and that scares even me - I've been back to handling dates and times again.

My experience is that dates and times are disproportionately risky for code. I have found more bugs in date handling code, even my own, than I care to admit. A colleague found a classic example of code that worked on Wednesdays (See: http://web.media.mit.edu/~lieber/Lieberary/Softviz/CACM-Debugging/Hairie...) and when we worked on Meet-O-Matic, we found at one stage that our code worked all months of the year except January. Obviously good testing is essential -- the point of these cases is that good testing can be very hard, as this code tends to be sensitive to environmental factors. For example, if you write code on a Wednesday, and it works on a Wednesday, you won't notice it's broken until it suddenly fails to behave on Thursday or Friday. Since no code has changed, this can be somewhat inexplicable.

Today I found another example. Some code that had worked for years suddenly started to hang. It was written by a former employee, probably about five years ago, and landed on my inbox. I had upgraded a whole bunch of CPAN modules, so it did seem likely that one of them was to blame, but there was nothing obvious.

Fortunately, it took very little time to find the culprit - some code iterated through days, using Date::Manip to add a day, until it matched the last day of a range. This code now started hanging on a day, specifically the 4th April. Yes, you may have guessed, it was the daylight savings transition. Adding a day at the DST transition only adds 23 hours, so you end up on the same day. If you then remove the time (which the code did) you end up on the same day, and there's the loop condition.

I don't really like Date::Manip. I use it for parsing crontabs, but for most purposes I use DateTime, which is slightly clearer on the object-orientedness. DateTime handles durations differently. A day is a day -- and does not need to be 24 hours long. If you add a day to midnight on the 4th April 2010, you get midnight on the 5th April 2010. This is different from Date::Manip, which gives you 11pm on the 4th April. (Date::Manip has "business day" logic which is similar, but also skips weekends, and we didn't want that.)

I like the intuitiveness of DateTime's handling of durations with subtlety. Adding a day (and adding a month, or a year) do not correspond to fixed multiples, but they are (usually) intuitive. What DateTime has done is bundle up all that highly complex intuitive logic, with all the nasty complexity of timezones and transitions, into a clear module which works well.

We still need to port the rest of our module to use DateTime, but using it for the straight math of adding a day broke out of the loop just fine. However, I'm feeling a Test::DateTime module would be helpful. One which would hack into the time handling in a way which would allow testing with different time environments, including a range of timezones, a range of years, and all months of the year/days of the week, and the usual start-and-ends-of-months. This would have caught this bug much sooner, as one of the reasons we'd missed it was that in testing the DST had corresponded with a weekend, and we were skipping weekends internally. Just trying a few different years we'd have caught the problem sooner.

And for the record, I'm not the only one that noticed this. It's covered at Date::Manip::Problems. I stand by the view that adding a day is not always the same as 24 hours. Personally, I'd make "+1d" and "+24h" behave differently, as conceptually, they are different.

One of the most annoying barriers to Perl portability is fork(). Let's face it, fork() is UNIX through and through. Windows doesn't do fork, you have to use some awful Win32::Process::CreateProcess incantation, and pass it the name of the executable you want.

So after last night's excellent Toronto Perl Mongers meet, we talk a bit about Plack. Plack is cool, Plack would do just what I needed if I could install the darned thing. Unfortunately, because of fork(), I can't.

I don't use ActiveState's Perl, or Strawberry Perl. I made my own, using MinGW, and I turned off threading, as I want it to run a bit faster. It does run faster, but it doesn't have fork() emulation, so all those modules that assume fork for testing (and it usually is testing) typically fail. Not many fall victim to this (WWW::Mechanize and its friends are most of them). Unfortunately, Plack is another one.

Part of me feels that a Perl API could actually allow something more like CreateProcess, which is pretty easy to emulate using fork() and exec(). The reverse is not true. Maybe I should just contribute a module which could replace Test::TCP which didn't need to fork to do its tests, maybe just passing a script file to a newly created child process.

In the meantime, please, if you are writing a module, don't assume everyone will have fork().

Let's be honest, Perl's threading model is not that good. In fact, it is for me the weakest area of the language. Generally, one of the main reasons I build and use my own Perl on Windows is so that I can remove it. And I remove it on UNIX too, but there I don't even miss it because at least I still have fork().

The issue cropped up recently on the Catalyst mailing list. Catalyst is often used with Apache, and can prefork a number of worker processes, using the threading system. In theory this is great: the code is parsed and loaded once, and then the worker processes are forked from it. The biggest drawback of not using threads on Windows is that each worker process needs to start from a clean interpreter, and this can take a good few seconds to get the application started. Doing this ten times over (if you have ten worker processes) is, to put it bluntly, crazy.

The fact that Windows benefits from fork() emulation is one thing: the issue is, should it be the same thing as threads. Threads are normally light, yet in Perl, using them essentially clones the entire interpreter state, not necessarily using the elegant copy-on-write semantics of a modern UNIX fork(). Threads are normally a good way of sharing stuff between processes, and even that is somewhat clunky in Perl.

Perl's use of 'multiplicity' is a great base - it allows multiple interpreter contexts, which really is the important bit. It would be nice to have that and a kind of object-based threading system that dispenses with the whole fork() emulation style threading crap. It would also be good for threads to have some OS basis, so that really they tapped into the benefits of whatever OS support you have -- that could be harder to deliver, but worth a try.

Sure, this means a little more programming and a little more care. Possibly even a little less portability. Probably better performance for the most part, but I'm more concerned that this runs counter the design approaches that Perl usually follows.

The thing about Perl is: it really taps into the underlying system -- that's the nature of the language. Threading seems to me a big false step -- it fakes a part of the underlying system to make systems appear more uniform that they actually are. That would be a Lisp- or Java-like approach, not a Perl-like approach.

As web applications go, the one I am developing is not lightning fast. It's not bad, and we have improved it, but you wouldn't want to scale it to tens of thousands of users as it stands. A small server is capable of handling maybe ten requests a second or so, which is fine for the kinds of client we have. However, we have hit a significant stumbling block at one particular client, who is getting much worse performance, and for no obvious reason.

First of all, things are a lot better than they were. When I joined the team, the "application" consisted entirely of Perl 5.005 style code, with no object-orientation, no web framework. It was basically a big, bad, and bloated CGI script. It was typically deployed as an ActiveState compiled executable, so Perl was being started for each request. Believe me, if you think it is bad now, you should have seen it then.

Now we use Catalyst. The old CGI parts are wrapped into a controller that handles some parts of the interface, buying us time while we migrate over to a proper framework. The old CGI layer wasn't completely bad - it had factored data access into a separate module. Unfortunately, it uses global variables everywhere, so there is very little opportunity to do caching or other performance enhancements. I'll come back to this later.

Catalyst is on the heavy side, but it is architecturally very nice. It gives us all the tools we need to contain the problem. With DBIx::Class we get an ORM which allows us to make the new code simpler and clearer. We use Template::Toolkit for most of the front end, which is very much nicer than generating HTML through the old CGI function calls. That code, the old formatting code for CGI, is truly awful. You have no idea how bad it actually is. Everything I was ever taught about how to write code is done wrongly there.

Anyway, back to the performance. We use the FastCGI engine, as most clients use Windows and IIS. Because we now have persistent worker processes, each worker process can typically deliver around 10 requests a second on a good-ish server, so with (say) 10 worker processes, you can easily handle a decent-sized department doing intensive stuff. Since the application is for browsing, searching, and analysing source code, this is pretty good.

So all was going well until we encountered one client. They wanted to run everything virtualized, so being cautious, we tested everything under VMware Server, and we lost a little performance, say 15-20%, but not a showstopper. We were not prepared for our application running consistently 3-4 times more slowly on their server compared to our VMware version.

The problem is a resilient one. We started by using basic web tests, which my colleagues like (I don't). They give a reasonable statistical picture, but hide the pattern, as they are all run through IIS, which makes profiling more or less impossible. We know a few things were a problem: McAfee was bad -- it was checking everything, even the temporary files created by the database we use. But McAfee is a giveaway, if you see "McShield" showing any CPU usage, you know it will be affecting performance. A few related tunings like this boosted performance by say, 25-30%. Useful, but nowhere near enough.

We also checked the system: they're running VMware ESX 3.5 with AMD chips, so we thought: AMD 64-bit systems can be an issue with ESX, but even using a 32-bit system makes no difference. Nor does adding memory, or processing cores.

So what about the SAN? Well, MySQL has a query cache, and even when the caches are full, the system is slow. Disk is not being waited on, at least not as far as we can tell, so the SAN doesn't seem to be an issue.

Networks? Nope, not them too. MySQL runs a little faster with a shared memory connection, maybe another 20-25%. Named pipes are also quicker than TCP/IP, but we're still slower. And, of course, our VMware Server baseline is also benefiting from all these improvements.

Perl's amazing Devel::NYTProf showed more detail. The big hit was in the database. DB queries were taking 3-4 times as long on their server compared to ours, oddly enough, even when the query cache is serving all requests and the disk isn't being hit. Now the MySQL query cache is trivial: if the string of the query matches, return the result. How can this be 3-4 times slower??

At this stage, if our application's old CGI data layer was better structured, and at least used parameters rather than globals, we could work around the problem with Memoize or local Perl caching. Even the query cache uses a round-trip to the DB server. Of course, we should probably do this anyway, but that is hardly the point.

I'm usually pretty good at debugging, and I have rarely been completely at a loss. This time, we do not have access to the ESX system, so we are about at the end of what we can do. There are reports that lack of memory affinity can be a problem, especially for in-memory databases (which is what the query cached MySQL is, in part). Both MySQL and our systems (being Perl) are memory intensive.

Having said that, most reports on the Internet are pretty dumb in their analysis, even frustratingly so. They usually say it is a disk latency problem - well, we know it isn't, as if I write a stored procedure and a Perl script doing the same set of queries, one runs 10 times faster than the other. The queries are the same, so it must be the communication between Perl and the database. And, of course, the difference is much smaller, maybe a factor of 4 or 5, on our VMware Server benchmark. Other explanations imply that two cores are bad because CPU scheduling is so hard -- sorry, I don't buy it, it is not that hard that it takes this much of your server.

I am sure we haven't seen the end of this problem yet, but in the process we have probably speeded our system by a factor of two, and maybe prioritizing architectural changes to add caching to Perl will double it again. It's all a worthwhile if rather frustrating exercise.