//-->

Last week, I released thevintageaviator.co.nz. It was probably one of the most challenging Drupal sites I've ever done, but also the best so far. It's a presentation of a local WW1 Warplane workshop, containing build stories, reference material, and thousands of great images of plane reproductions and archive materials.

The build story is a long one, mostly because the requirements shifted along the way quite a bit. This story is targeted at current Drupal site developers who are interested in the tools and techniques used. Mostly to head off the "so how did you do that?" questions we get whenever we do a write-up :-)

However among the interesting thing for developers and designers will be the complex layered look, the way this layout was optimized to scale to all screen sizes, and the surgery I did to Drupal to make this all happen.

Initial feedback we've received so far has been overwhelmingly positive (save some justified grumbles about the page weight) and I'd like to share this case study.

Build Process

The development environment went through many stages. Syncing a working database with working development is not a job for the faint-hearted.

For almost all of the project, the tech side was me and only me. Plus one Graphic Guy, and a part-time wordsmith. Over a year an a half, mostly in small, concentrated bursts with long gaps.

In that time, the theme and purpose of the site went through many changes. I won't go into detail, but at one point (version 15) I actually made a clean break and rebuilt from scratch, transferring the content by hand and with views_bonus_export, node_import into a new sitemap, theme and architecture, and re-importing the galleries (image_import) into their new (named) locations. That was a good and necessary move! Once you've made enough mistakes, you've gotta know when to start with a clean slate!

Due to the extended build period (and to our own fluctuating circumstances) the site moved around a lot. First locally on XAMPP under my desk at the office, mirrored (working(!) on a thumbstick for offsite work). Then I took it home and ran off my local dev machine (Ubuntu). When we got to preview stage I pushed it up to a hosted server (Mediatemple) we could all get at a bit easier (My ISP wasn't really up to serving from home).
Then we shifted office and I had the Dev site running on a real Ubuntu server, while still using the hosted one for content development.
Eventually we could push it to its intended destination on a local helpful hosting provider (http://face.co.nz) but that was PHP4 only (some rework required, small stuff).
And once that went live, the content requests just crushed that machine (the site is HUGE file-wise), and they immediately offloaded it to a dedicated box. Which I was able to upgrade to PHP5 again in the process.
If nothing else, this was stress-testing for platform compatability!

Development environment throughout was The Eclipse IDE with PHPEclipse, The Web Dev toolkit, and occasionally RDE (which allows remote-site project editing over SSH etc). I really like Eclipse.
And on Windows as the desktop (plus a little from my macbook). Firefox (with of course the web-dev toolbar) for most dev, with frequent IE6/7 switches (and its developers toolbar, which isn't as cool, but still helps heaps).

Content management continues on the live site, while remaining code and theme development happens on our office dev box.
As you can imagine, keeping that all in sync was a tedious undertaking. I used and improved some scripts and tools I'd developed earlier to assist in site synchronization : sitesynch.module as well as a bunch of commandline shortcuts (rsync and mysqldump)

Standard Modules

The site is Drupal 5 (now 5.7). It was first scoped and prototyped a year and a half ago. Even now in the middle of 2008 I don't think I could get the right results up and going in D6, although I'm making an effort to do so with current projects, and intend to drag any medium-sized modules I find lacking into the brave new world.

The Usual suspects

CCK of course, although I avoided it for almost the whole development, and only pulled it in at the end to add a credits field to the images.

Views I use mainly for some admin screens, like a 'show all content' all-in-one audit page for proofreading. I also use a fixed (modified for table layout, taxonomy terms and image thumbnails) version of edit_view for bulk image caption editing, as managing thousands of pictures with several clicks each is painful.

coder and devel I kept in the background during development.

Other standards include:

• site_map.module (mostly unmodified, just minor theming)
• xmlsitemap.module (worked out of the box, which made me and Google happy)
• nodeteaser.module for custom intro text, as the first paragraphs were not helpful enough in general, and the design really constrained the size of the text we could use.
• htmlcorrector.module ... because there's a WYSIWYG :(
• dhtml_menu does the expanding main menu, although I had to break it a bit to change the behaviours where it was 'unintuative' or redundant when clicks didn't actually go anywhere. I believe this could have been solved through design (showing an expand/contract icon) but that never came through. It still needs to be broken a little more, as currently it loads the full menu tree even when not required. Earlier I tried the jstools activemenu also, but the speed/efficiency tradeoff for AJAX didn't work out for me.
• taxonomy_multi_edit.module for some admin stuff during development, shifting images between galleries etc. I can probably remove that again now.
• pathauto.module - pretty standard stuff, although I'm not sure it's working every time.
• sifr.module for the page header font. It causes a little page flicker however, but that's something to do with margins in the themes. It does show up a little slow on some connections, which is disconcerting, but the typography won out.
• fckeditor.module We had to have a WYSIWYG, even though I find them all various flavours of evil. Once I and the content editors had got used to the pain, and I'd inserted a handful of custom styles and stylesheets for it... It's OK. After much cursing, our designer was able to do some pretty cool things with the type.
• imce.module For image insertion. The file browser is horrid, but it works. Navigating thousands of images over a net connection is never going to be easy, but after I'd ranted about the importance of good file name conventions, and added tokens to create custom directory paths it became almost manageable.

Image Galleries

Choosing an image handling solution for Drupal is always a challenge, because there's so many of them.
I choose to stick with the simple ones I know and understand, so I did not use any of the fancy 3rd-party gallery extensions, just image.module and image_gallery.module. Importantly, I always think of my resources (images) as first-class citizens, with their own metadata, tagging, and place in the content management. For this reason I couldn't use any method that did not treat images as nodes. This eliminated the attachment-based ones.

We worked up a themed combination of fixed, proportional and floated displays for different contexts, as well as several extra additions (see below). Basically, there were many iterations and many styles before we got to this styled solution.

I worked in an image gallery ordering option, so I could list by filename-alpha (which was just a requirement for bulk-uploaded galleries, reverse by date just was wrong). Image+views came along later, and makes this option slightly less neccessary.

I used image_attach.module to select a representative image for each story in the teaser, and taxonomy_image.module (which Nancy has been steadily improving recently) for the topics.
Both slightly modified over time to integrate with imagecache.module for the custom thumbnails (more on that later).
I considered imagefield for that, but it wasn't mature enough at the time, and (I think) didn't create image nodes, just 'fields' to do it. And image_attach, although it's being left behind a bit now, just felt more integrated into the old Drupal UI and theme system.

I modified image_attach so it could re-use existing images from our library, rather than requiring a fresh upload (and probably duplication) each time. The drop-down list is huge, but with good naming conventions it's still OK for us so far.Hm, has that not been contributed back yet? Or maybe someone else did the same recently. Ah well, it's there now.

Modified Modules

• taxonomy_context.module

  Trying to display taxonomy-ordered structure, pages, teasers, and menus and breadcrumbs still worked out much harder than it should be. I worked with taxonomy_menu for a while but just found it incompatible with too much of the normal Drupal-way of doing things, and the URLs were horrible.
  For a while I used my own version of a taxonomy_menu module that rebuilt menus on-the-fly every time taxonomy terms were modified, but that got too complex. As it is, we still add items to the menu manually as needed, for control. With some help from my updated term_edit.module.
  I hacked taxonomy_context a lot to theme it and get it to display items in the structure I wanted (it was severely lacking in good theme hooks, and still pushes 'content' into the 'help' zone!) and to support recursion.

• video.module

  Although it worked fine in prototyping, we ended up making a totally custom themed FLV player, so I combined video.module with a new theme_video_play_flash() override.

• imagecache.module

  ... I did a lot of things to imagecache. A big shout-out to dopry, Dimm and team for the work done there, especially the move into imagecache-2, which happened over the course of my work.

  First, I glued it into image.module presets - So I can now use imagecache presets (a nicer scale&crop, and watermarking) on normal image.module derivatives like 'thumbnail' and 'preview'. That kept everything nice and Drupally and together!

  Then I worked through the watermarking and extended color effects, first in imagecache-1, then imagecache-2, and now in its own contrib module : imagecache_actions.module. This was used over time to create many different effects for our graphic looks - only a few of which have ended up in use in todays site. Currently, its main job is watermarking.

• lightbox2.module

  I severely hacked lightbox.module. Our current version is quite customized to please the designer and the site theme. As all lightbox code is written on-the-fly via javascript, there was some hairy code and javascript changes buried in there. But after a week or two of reworks (because of several totally different designs, arrg) we got it stable again. What we have now is actually one of the closest to the original ... after all that :-}.

  Later, I reverse-engineered a little more of the lightbox.js to launch it on demand from within a Flash image flipbook. Gee, that was fun. Try it here.

• piclens.module

  The new guy on the scene, and I love it! DO try piclens! I fell in love with it, considered doing a module, then three days later saw swentel had a working implimentation ready for me. Thanks!

  Although we are not publishing the piclens slideshow viewer directly, our galleries are piclens-feed enabled (with many experimental tweaks, like 'next:' links). PLUS, once I discovered how simply right the piclens/RSS feed method was, we reworked and developed all the flash gallery extras to read the gallery feeds! Versatile, re-usable, semantic, easy, portable... I rave!
  I think I'll be using this methodology for ALL my client-server communication from now on.

Custom Modules

Note: Most of these modules are not yet contributed back to Drupal.org. Some may, but please don't bug me about it unless you want code you may cut yourself with.

• edit_term.module

  I put together a handful of enhancements to term management pages and we bundled that into this module. I don't know how we used to manage without it. It saves clicks! It's a few small changes that makes Drupal a better UI. This project IS available for download

• imagecaption.module / lightbox_everything.module

  I took a look at imagecaption.module, then hacked it beyond recognition and came up with something quite different. This is a filter which finds raw images embedded in a page (as in via the WYSIWYG) then backtracks to discover what (if any) node this image represents.
  It then reformats the embedded image to add a few classes and css stuff (for alignment and scale) and adds the caption, border and hooks for the lightbox and imagecorner effects.
  Pretty tricky all 'round.

  It MEANS that we can use the WYSIWYG to just dump images into a page, and they end up looking like you see here, resized to the screen, and lightbox-enabled.

• gallery_attach.module

  After a few attempts at deciding whether our stories should have all their images embedded in the text, or link to the full gallery, or as a child page, or a combination, I came up with gallery_attach.module.
  This in the first instance works a bit like image_attach. On the node edit screen you choose an available gallery to 'attach' as supporting illustrations. During development I extended this from the basic "show all thumbnails below the story" to add a "carousel/preview" view, a "flowview", and a "flipbook" view. And eventually, an arbitrary block view to add anywhere, and a gallery-attach-block-filter so these views can be embedded in the text also. The code is cool, but unpolished. The gallery feed data draws somewhat on the piclens feed mentioned above.

  By using this utility, I was also able in some cases to create page nodes that contain nothing but a gallery thumbnail view (indistinguishable from a normal image/tid/n page) that was easier to work with in the CMS, and could have a descriptive paragraph, term and teaser assigned to it. This is easier and more consistant than trying to tag and position real gallery views.

• spacefiller.module

  This is a custom module I wrote to leverage the jstools dynamicload(block). It incrementally requests content from a pool of (imagecache-processed) gallery images to fill up gaps in the page. It does this post-load, as it's only then that we can calculate the page dimensions, and also so that requesting all those images doesn't slow down page loads.You'll see it in the left column on a huge page

• phraselinker.module

  I know that hyperlinks are easy, but even (especially?) with a WYSIWYG, finding the target you want to link to is a chore. I know there are a few existing markdown filters (wiki) to make it easier, and probably a few more bite of this apple (like glossary.module?), but still...

  What I did was add an extra keyword filter to auto-link every occurance of a keyphrase to an associated URL. Now some pages are liberally scattered with cross-references automatically, making some of them look a bit like wikipedia or Everything2.
  It's a cheap win so far, but I'm not sure how this trick will scale. So far we just have to be careful to not add too many generic phrases to the list, and ignore the fact that sometimes it gets a bit self-referential. (Pages don't link to themselves, but they are very likely to link to parents liberally.)
  It was a fun module to write, but it needs some better tuning to be safe in the wild.

• mediadescriber.module

  You won't see this in action on the site, it's an admin/archiving utility I added to help manage image galleries. Once I found myself dealing with hundreds of captioned images (supplied on CD, as uploading individually was impossible) and changes to them requiring exports, dumps and replacements, I started using a simple metadata file - descript.ion text files (as seen in ACDSee and others) to read the file info in. And, soon, to write it out again alongside my image galleries as they are stored on the server. Based on experience with site migrations, I hate my cannonic info to be tied in to any one system or database, even Drupal.
  This tool lets me save, archive, zip and send full image galleries with their titles and captions in a format I know will still be useful in a decade. But for now it just meant I could communicate meaningfully with the photographers and indexers who were supplying the content I needed to manage.
  It integrates directly with image_import.module when reading, and saves to a text file when updating images.

• sitesynch.module

  I mentioned this earlier. It's a tool I developed over several sites to provide one-click 'snapshots' of code, files and database from any dev site, and also uses RPC calls and other tricks to download, unpack and sync between two mirror sites. I'm also developing page-compare utilities to assist in one-click one-page 'updates' from site A to site B. In Drupal 4.7 I even had it working to directly compare any Drupal admin form with the same page on a remote site and submit changes to it - useful for syncing admin changes I'd made on dev to the staging server. Unfortunately, the Drupal5 API with its extra security checks broke that feature, and I've not been able to upgrade to that again.If that sound interesting to you, sorry, it's got too many dangerous corners to be released yet, but it may happen.

• pathfinder.module

  Another tool to make migration less painful.
  Pathfinder is an URL-rewriter that works both as an input and output filter.
  When developing content on one site that's going to end up on another host, and trying to embed images or paste links ... it's easy to accidentally find yourself with full URLs that just won't work when live. Pathfinder scans links on submission, and modifies the source before it even gets saved to remove full local http: links and other site specific (eg drupal-in-subdirectory) artifacts. At the other end, it has an output filter that checks for some common errors and repairs them. In my case I tuned it to detect missing embedded images and find the real location if possible. Neccessary when I applied my grand file-renaming patch.
  Now I (and more importantly my content editors) can copy and paste URLs with abandon, knowing that those landmines are being diffused when I click 'save' and won't suddenly bite me when I switch from test.example.com to live.example.com!This one is a candidate for drupal.org release, although it's highly tuned to my development style and requirements. I'm sure it will need more exposed options (which will be complicated) before it's much use to other folk. If I can get it workable, I think it will avoid a lot of newbie gotchas.

I apologize if this list is a bit of a tease, but no, you can't have them in the forseeable future. It's just here as a run-down of the challenges I encountered and how I solved them. I don't have enough unpaid time to commit to more support requests.

Theme customizations

Sorry... there is one table holding the layout together.

Although I'm a semantic snob, by the time we got to this (Template version 17) with its transparent, overlapping corners, tiled and positioned backgrounds, Fully liquid displays and everything... Pure-CSS just couldn't cut it. Not with reasonable support for IE6.
So there's one old-fashioned table there to hold the corners out, although all the imagery is pure-CSS, and switchable too! There was at least one other 'page' background with different textures and colors that uses the exact same HTML hiding within my theme dir.Floated CSS divs just had no concept of 'align:bottom' or 'height=100%' for the elements I needed.

Visuals

The gorgeous design is all down to my man Mark Williams here at http://gomi.co.nz/ . He put up with my curses against Flash and drop-shadows while I put up with his always putting elements outside of the rectangular spaces I'd allocated for them.

Stand-out achievements include:

• Finding a way to get full floating alpha-transparency working in all browsers (even IE6) and at the same time reducing page load by like 400K over what it was taking using PNGs!
• Coming up with a dozen radically different designs over the course of the project, each of them perfect in their own way, and still ending up with this magnificent look.
• Jumping on the boat with the semantic RSS feed concept and developing our custom page flippers in a way that makes my semantic pedantics rejoice at the thought of using Flash sensibly!
• Check out what happens when you resize the banner on the front page! I actually got this effect working first with pure-css and gifs first (!) but this is glorious.
• Of course, all of the image-prep and effects, using Photoshop to do his thing but still leaving layers in a way I could slice transparencies out of later.

In broader Kudos:

• The image resources we had to work from were outstanding! Both the modern photography and the scanned archive imagery was of the highest quality, and inspiring source material. Our client was very forthcoming with sourcing and preparing these for us, with some fast turn-around from "what if we had these sort of images to work with?" to "Here's a CD with 60 images to choose from!". That phase was great.We have a library of goodies to be prepared to add even more texture and variation to the pages as we go on.
• Our client-end content manager had to deal with a truly miserable internet connection and a small-screen laptop for much of the build. Trying to preview, and upload huge images - using the standard Drupal interface - was tiring.
  Heaps of respect to Gene, our brand-new webmaster and his newfound ability for finding interesting captions for hundreds of photos, and his patience in believing me when I made excuses for stupid browser behaviour. "Have you tried a hard refresh? I just changed the CSS again!"

Tricks & Unique stuff

Throughout the process, a driving requirement was that pages should fill the screen at all times. No fixed-width here. And have no empty space. And the screen it was being previewed on was a maximized HD widescreen running IE6 (!). Yet I needed it to be actually usable for human beings... and our content editor was on a 1024x768 laptop. Tricky.

Resizable layout

After much work, I got our thumbnail resizing magic happening so that in every resolution, our page layout looks as near as can be imagined to a properly formatted magazine layout. Lots of fun, lots of pain and testing. Fully degradable to no-script, no-flash, no-images (although I gave up on some esoteric effects for IE6).

High-res

The source images for our site are huge, high-quality, and they just kept coming. The file directory is 2.5 GB so far! We needed to have the 'view-full' images really worthwhile, plus the watermark changed a few times, plus there's a possibility we want super-high res zooms for the archive photos, so there's a backup of the full res originals indexed also.
What this means is that backups and transfers ... and especially site synchronization - could take ages. Learning how to do rsync effectively was vital. As was having decent SSH access to my hosts and ssh-key support. This tool let me only update changes that NEEDED to change, not just blindly copying an FTP dir and crossing my fingers. I also often configured it to ignore my derivative images when I didn't really want all of them to update due to a small watermark change just yet.

Clutter

Our margins are variable, our page heights are variable, but we wanted no gaps. With a bit (lot) of jquery magic (spacefiller.module), and some help from dynamicload.module, you will see that the margin gets decorative content inserted according to page size every time.

Our decorative elements are all managed through the CMS (as specially tagged images or custom content types) so they can be managed via normal node creation, and extracted with views.The 'clutter' elements scattered on the 'desk', the topic banners, and the side images are all handled this way. It's a little bit more complex than just custom-coding some CSS or PHP directory scanning, but has paid off and I'm sure will help as the site grows.

The clutter is mostly placed using views blocks (list views), but unthemed a lot to remove 90% of the markup. It was tricky finding exactly the right theme function names to over-ride, but eventually I extracted them from the views theme_wizard and discarded all the wizard-ness.

Yes, half the image elements are Flash! WTF?
Basically, we had them in PNGs for most of the duration, but when I found I was looking at 1.4 MEGs per page load I screamed. Mark declared Flash was better at compression and did full transparency with no problems. I said "Prove it". And he did. It dropped many of our lovely photo-graphics from 200K to 20K, and indeed layered safely, even providing a win on IE6. I conceded, and we got a result that works! It's all non-sematic stuff anyway, so I was content.

Image Borders

No, that's not done with image processing. Those image corners are layed on with some css (and jquery) magic. Partially as CSS Sprites where I could.
At one point I did get imagecache to do the job, but I had to drop that because it looked silly when scaled. After I got that down, I was (mostly) able to add the right bits to the theme and use this same effect and script throughout the site for taxonomy_image, image_attach, image_gallery, imce and teasers! This did require heavy theme overrides in places, but it's working.

Image Menus

Using a combination of CSS sprites for the hover effect, and some fancy theme code (theme_menu_item, theme_menu_item_link) and a small script that looks for pre-prepared images in a resource directory named according to menu text the main navigation has groovy text. CSS tricks are supposed to ensure that it degrades to plaintext OK, but the jury is still out on exactly the best image-replacement technique.

Short-cuts

Just to speed things up, for editors, images get a small [edit] button overlayed so they can hop straight in and update captions and make correctins from where they see them. It looks pretty trivial, but was really neccessary once lightbox started taking over all the clicks. This is done in the theme (because that was quick and easy) adding an node/n/edit link, plus a little style.

As mentioned above, I used also editview to display a mini-version of the image node edit forms. This was handy for some bulk updates.

CSS

Early on in the last remake I tried using a reset.css to see if that would reduce my cross-browser pains. It did and it didn't. I'm not sure I'd use it again.

In development, I tried breaking off several of the tasks into their own stylesheets, often custom ones named after the module it was to affect. That was mostly a win, but got tricky sometimes when deciding where some directives should go.
It did make some of the tasks easier, eg all the menu theming in menu.css, and the gallery frames in image_gallery.css. More formality, slightly more complexity, possibly better maintainability, and certainly better for change control.

The theme you see here (vintagedesk) is based on almost no available theme I can remember. I tried zen in one iteration, but it just broke a lot of the methods I was used to from Drupal theming. I think this was started from cleanslate or something similar, as that far down the track I knew I had to do 90% from scratch.No, it will not be released. I hope you understand :-)

Lessons Learnt
Stress-test

I proposed a quiet launch to give the search engines time to seed before announcing the site. I went live mid-week and started submitting and remote testing. A friend of a friend got a link, posted it in a forum, and it entered the blogosphere.
The next morning, precisely when google decided to start downloading my huge images, and some dweeb decided to run a bad-behaviour HTTrack mirror on all of out 2.5 GB, people started finding it, and not getting in because the server was unhappy. Those pages are heavy, I know, but it seems it was the sheer bulk of the images and videos that took it down to a crawl. Having a mega-connection actually worked against us, as the suction from the available pipe was bigger than the Apache server could keep up with. Or something like that. Drupal/PHP/SQL performance was also probably doing the first cache build of thousands of pages of requests, as I'd dropped the cache at transfer time, but not spidered it myself. So that did slow it down too.

I blocked the HTTrack robot, but our host got a scare. Thankfully they immediately enacted a couple of Apache throttles, and by close of day we had co-ordinated a transfer to a dedicated server (it was taking down its neighbours). By that time, pretty much, the bot-fight had settled down, but it was a non-optimal user experience for those that got the leaked URL. Of course those folk were some of the inner circle, so that was a bad look.
Basically, by the time everything else was done, I didn't have a contingency or scaling plan in place. :-(
Partially because the scope of the site had changed radically into a media server from the brochureware we had planned, and I'd not re-evaluated requirements since.

Quick tips:

• Make that redirect from www.example.com to example.com early - before you open the site up to spiders or start taking logs. Or else referrers will fill up with crap
• Do warn your hosting company that the site you are going to make live is going to be 100x bigger than what you told them it was going to be when you signed up for the contract.
• Figure out a scalable method of storing images and files as soon as you can. Just dropping them all in one big folder just doesn't work beyond a few hundred. User quotas were not answer for me. The worst bit was breaking everything (all embedded image references) and having already-proofed pages fall apart following this global change. You need some serious fu to fix that in a hurry.
• (Something I didn't do) Figure in some process flow for checkin, proofing, sign-off, design review of your pages. I know there's a few taskflow modules around, but I never found the right time to turn it on and learn. I enabled 'revisions' at the half-way point when our editors multiplied, but never really made use of them.

• Check your printable pages each time the design changes seriously. I've been pretty good about this, but not exhaustive. And basically it was mostly turning things OFF in print.css. Given a spare day and lots of scrap paper I'd like to start ADDING some print-only effects like a B/W letterhead. If you don't already know it, good printable page breaks in browsers are still freaking hard to do right, even with the available CSS 'hints'.

Disclaimer

Yes, you are now free to poke around the code and tell me what I could have done better. The CSS (cached version) is probably not much fun to look at, and I know there's some confusion in there, but we had many revisions and quirks layered on, so not much of it can be held up as a shining example to work from. You are better off using firebug.I know it's graphic/pageload heavy the first visit, but CSS/style caching makes up for that pretty quick. I was constantly warning about page sizes, but the mandate was simply "Don't compromise just for old machines, this is a design for next years computers". So there!

In the pipeline is probably some performance optimization (which I've done before with APC) and improvements on that side (I got bitten, as described above), and there's a small list of 'phase 2' tweaks yet to make, like RSS promotion and ongoing newsletter-style archives.

Dan Morrison, (dman) @ Gomi

Drupal version: Drupal 5.x