Monday, March 5, 2012

The joy of JAX-B and SAX

I decided to post this after an interesting journey to build an XML parsing component that would allow me to produce really good validation messages, better during deserialization but also for post-processing those kinds of rules you can't express in schema. So, typically if I want to get good error/validation with location information I would normally reach for SAX the Simple API for XML and build a ContentHandler to handle the events from the SAX parser. In Java I can ask for the setDocumentLocator method to provide a locator object that allows me to determine the line/column in the XML content at any time during my handler.

So, what's the problem? well two things, I cannot use this generically as each content handler has to be really coded to the type I want to deserialize, and SAX handlers can get complex to write and maintain. So, what about JAX-B which provides a nice simple API to deserialize XML directly into a Java object model? Well the problem is that while the implementation does usually provide location information for serialization errors not all errors that could be caught by JAX-B are caught and so we need to post-process and at that time we no longer have any location information, just our object model. To gather up all the errors generated by JAX-B we need to provide an implementation of the ValidationEventHandler and pass it to our unmarshaller using the method setEventHandler. Our implementation is pretty simple (and a commonly documented pattern) and only records all validation events into a list, ensuring that the handleEvent method always returns false or the parser will assume the error is fatal and stop parsing. Note that we actually transform the JAX-B ValidationEvent into our own ValidationError class, more on this later.

So, now we have captured all the validation events we need somehow to persist the line/column information for our object model so that when we do additional validation/consistency checking we can provide good, meaningful errors. Ideally we want to be able to record the location information outside the object model, we don't want to have to add a line and column field to each model object. Our solution is simple, we keep a Map of parsed object to Location so that we know the line/column of the start of the XML element that was used to deserialize the object. But how? After all JAX-B is pretty much a black-box, we ask it to deserialize an input source and we give it the expected type of the root object. For a start JAX-B does allow you to peek into the process via the setListener method which takes an instance of the Listener interface and allows you to take action on beforeUnmarshal and afterUnmarshal events. Interestingly though JAX-B has a mechanism whereby you can actually retrieve from your Unmarshaller the internal SAX handler used by JAX-B and then use the SAX API to drive the deserialization. The advantage of this is that we can wrap the JAX-B handler in our own handler before passing it to the SAX parser, overriding any SAX events we want. In actual fact we only need to override the setDocumentLocator call and keep a reference to the Locator object. The resulting class, DelegatingHandlerImpl extends the JAX-B Listener and implements the JAX-B UnmarshallerHandler (which in turn extends the SAX ContentHandler); the code for our handler is shown below and basically by using the before unmarshal method and the SAX locator we can build an internal map that tracks the location of each parsed object.
class DelegatingHandlerImpl extends Listener implements UnmarshallerHandler {

    private final UnmarshallerHandler unmarshallerHandler;
    private Locator locator;
    private final Map<Object, LocationImpl> locationMap;
    
    /**
     * {@inheritDoc}
     */
    @Override
    public void setDocumentLocator(Locator locator) {
        this.locator = locator;
        this.unmarshallerHandler.setDocumentLocator(locator);
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public void beforeUnmarshal(Object target, Object parent) {
        super.beforeUnmarshal(target, parent);
        if (target != null && this.locator != null) {
            this.locationMap.put(target, new LocationImpl(this.locator.getLineNumber(), this.locator.getColumnNumber()));
        }
    }

The parser code is shown below, it is more complex than you would expect to see if you are used to SAX and certainly more than you would expect of JAX-B but mostly we're just setting up the JAX-B/SAX combination and plugging in our two additional classes, the DelegatingHandlerImpl (line 21) and ValidationEventHandlerImpl (line 24). We construct both the list of errors (line 4) and the object to location map (line 5) in the parser so that we can make them available to the caller after parsing is complete.

    public T parse(final InputSource input, final String schemaPath, final Class classOfT) 
            throws ParserConfigurationException, IOException {
        this.result = null;
        this.events = new LinkedList<ValidationError>();
        this.locationMap = new HashMap<Object, LocationImpl>();
        try {
            
            // Standard JAX-B
            final JAXBContext context = JAXBContext.newInstance(classOfT);
            final Unmarshaller unmarshaller = context.createUnmarshaller();
            // Setup schema validation if required
            if (schemaPath != null) {
                final SchemaFactory sf = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
                final InputStream schemaIS = ClassLoader.getSystemResourceAsStream(schemaPath);
                final Schema schema = sf.newSchema(new StreamSource(schemaIS));
                unmarshaller.setSchema(schema);
            }
            // Now retrieve the SAX handler that JAX-B uses
            final UnmarshallerHandler unmarshallerHandler = unmarshaller.getUnmarshallerHandler();
            // Wrap it in our own handler
            final DelegatingHandlerImpl actualHandler = new DelegatingHandlerImpl(unmarshallerHandler, this.locationMap);
    
            // Now create and add an error handler
            final ValidationEventHandlerImpl errorHandler = new ValidationEventHandlerImpl(this.events);
            unmarshaller.setEventHandler(errorHandler);
            // Add a listener for before/after unmarshall events
            unmarshaller.setListener(actualHandler);
    
            // Now setup SAX
            final SAXParserFactory spf = SAXParserFactory.newInstance();
            spf.setNamespaceAware(true);
            
            // Start the SAX parser but using *our* new handler
            final XMLReader xmlReader = spf.newSAXParser().getXMLReader();
            xmlReader.setContentHandler(actualHandler);
            xmlReader.parse(input);

            // Retrieve the result from the handler, note that this is actually
            // the bridge back to JAX-B
            this.result = (T)unmarshallerHandler.getResult();
            
        } catch (UnmarshalException ex) {
            // ignore, these are reported in the validation errors.
        } catch (JAXBException ex) {
            this.events.add(new ValidationErrorImpl(Severity.FATAL, "JAX-B configuration exception", ex));
        } catch (SAXException ex) {
            this.events.add(new ValidationErrorImpl(Severity.FATAL, "IO error reading from InputSource", ex));
        }
        
        return this.result;
    }

I've posted all the source for the parser implementation as well as some basic (for now) test cases on Google Code in the validating-jaxb project.

Friday, February 3, 2012

Amazon S3, more than a blob store

Having built a number of RESTful services I have developed the storage for resources a number of times a number of different ways. It's important to realize that there is a progression that the first few projects go through until you work out the patterns that really work. The easy ones are obvious, keep the HTTP methods simple, avoid adding crazy request/response headers, use ETags, if you need content negotiation only use the standard Accept/Content-Type mechanism. The fun starts when you try and wrap your head around updates to resources, and the easiest way to get yourself out of the spiral discussion of partial updates, support for PATCH and all manner of other craziness is to treat all resources as technically immutable and every PUT to the resource creates a new version of the resource. This works well as the version identifier becomes a very stable ETag and with a little effort you can expose a URI for each version allowing you to look back and forward through the history of a resource. For a lot of projects I've used the resource store to store temporary resources (sometimes as a queue, the output from process one is put into the store to be picked up by process two) and it's annoying to keep having to write another "reaper" process to remove expired resources.

Recent changes to the Amazon S3 service have really made this a viable resource store, it's not just an online file store any more. While S3 has supported metadata associated with objects and a great API for query it's the additions to the API recently that

  • Versioning, it's been there a while actually but not many people use it. Basically you can turn versioning on bucket-by-bucket to store all versions of every object in the bucket. The query API is now extended so as well as listing all objects in a bucket you can now list all the versions of an object.
  • Expiration, you can now specify an expiration time for an object and S3 will automatically remove the object. 
  • Multi-Delete, if you have to still perform your own clean-up operation you can now query for all "expired" or otherwise superfluous resources and delete a set of resources in a single API call to S3.
This makes S3 a much more complete solution and certainly I hope I don't have to write another RESTful resource store any time soon, at least anything more than a domain wrapper around S3.



Tuesday, December 6, 2011

Guernsey - a REST client for Python

Having written a number of REST clients in Python over the last few years, they all start out the same way; write a simple piece of code to call one service, then realize that you have to call a second service, then a third and so the code becomes more and more generic. However, starting the next project I have to untangle that code from project A and copy into project B - if possible, changing an employer does make code reuse a pain.

So this time, when the need arose I decided to write the client as a separate and reusable package - Guernsey. I then looked at the API I had developed before and it really wasn't very good, or rather it still showed aspects of the first project I created it for. Right now my day job also includes writing Java REST services and we use JAX-RS which give us a really annotation based model. As an implementation we use the Sun Jersey implementation which also has a very nice client package, so I set out to see if I could use that design as a basis for a Python package.

The result is pretty close to the Jersey API, which may be of value to anyone working in Java and Python (which I do regularly) but also it simply benefits from the effort someone else put into the API design. I also took care with naming to keep things as close as possible, but to use a Python style (lowerCamelCase became lower_camel_case, some getter/setters became simple object properties, etc.).

The following example shows a simple client that accesses a public REST service.

from guernsey import Client

client = Client.create()
resource = client.resource('http://www.thomas-bayer.com/sqlrest/')
resource.accept('*/xml')
namespaces = resource.get()
  1. The Client class is the entry point for most common cases, so for most uses you can simply import the one class.
  2. Calling Client.create will construct a new Client object, this method takes a dictionary that provides configuration parameters, or without configuration will create a new object with all default values.
  3. Calling client.resource will construct a new WebResource object which is the actual object you perform REST methods against.
  4. The WebResource object has a number of methods which can be used to configure the resource behavior; in this example we simply set the accept headers to denote that we wish to be returned XML data if possible.
  5. Finally we call the get method on the WebResource (each WebResource supports get, head, put, post, delete and options by default but could be extended to support other methods).
All the configuration methods on the return WebResource the object back to the caller which allows configuration methods to be chained together which can make some code more readable.

namespaces = client.resource('http://www.thomas-bayer.com/sqlrest/') \
                   .accept('*/xml', 0.8) \
                   .accept('*/json', 0.2) \
                   .accept_language('en-US') \
                   .get()

In this example we can see that the setup methods accept and accept_language are all chained together and finally the get method is called. A number of simple methods are provided to manipulate common headers, although if you want to add your own header you can use the add_header method.

Navigating Resources

Where your REST service follows a true hypertext (HATEOS) style one would expect that navigating between resources should be simple as any resource would provide within it a navigable URL from it to any related resource. However, some services still do not provide a true hypertext style, either returning relative URLs or worse returning identifiers which a client has to use to construct a URL.

Guernsey should help you in all three cases:

namespaces = resource.get()
data = namespaces.parsed_entity
# Hypertext link:
new_resource = client.resource(data['resource_url'])
# Relative path:
new_resource = resource.path(data['resource_path'])
# A sub-resource identifier:
new_resource = resource.sub_resource(data['resource_id'])
# Just an identifier:
new_resource = client.resource('http://example.com/{resource_id}', data)
  • In the first case we have, in our parsed response data, a full URL and so we can easily call the client.resource method to construct a new WebResource.
  • In the second case we have a new path relative to the path of the current resource, in this case we can use the resource.path method which will construct a new absolute URL by resolving the provided path against the current URL.
  • In the third case we only have, in our parsed response data, an identifier which corresponds to a path segment that can be appended to the current resource path (commonly termed a sub-resource identifier).
  • In the last case we need to construct an entirely new resource, but using the identifier in the parsed response data, in this case we use a templated resource URL in the call to client.resource.

Handling Entities

In the section above we mentioned the use of a parsed entity, the Guernsey client has the ability to detect a known resource representation and then convert from its serialized form returned from the REST service into a common Python form. In all cases any entity returned from the service will be put into the entity property of the response, if the Content-Type header in the response identifies a type that can be de-serialized then the response will contain a parsed_entity property.

namespaces = resource.get()
raw_data = namespaces.entity
xml_data = namespaces.parsed_entity

The client performs this by using the EntityReader and EntityWriter classes in the guernsey.entities module. These classes define a simple interface for a subclass to determine whether a given type can be serialized, and if so to convert it to/from a reasonable Python form. Instances of these classes are added to the array property entity_classes on the Client object and will be called in order to process any request and response.

The guernsey.entities module contains reader/writer classes for XML and for JSON and these are added by default to the standard Client. The JSON reader will convert serialized JSON into Python dictionaries and standard types using the standard json module, the writer will serialize Python dictionaries and standard types into the JSON serialized form. The XML reader and writer convert between serialized XML and ElementTree objects from the xml.etree.ElementTree standard module.

Additional entity readers and writers can easily be constructed, and if added to the client will be used by any resource created by the client.

Client Filters

An additional capability of Jersey, and therefore Guernsey, is the ability to write client-side request/response filters. These filters work much like Servlet filters in Java or WSGI middleware in Python in that they are chained together, the handler is given the current request and is able to modify the request before passing it to the next handler in the chain. The last handler actually performs the request, constructs a response object and returns it back down the chain. Each handler is then able to modify the response before returning it itself.

A number of common filters are provided to enable logging, Gzip encoding and MD5 content hashes. These can be added simply by using the add_filter method on a resource.

resource = client.resource('http://www.thomas-bayer.com/sqlrest/')
namespaces.add_filter(LoggingFilter('TestFilterLogging'))
response = namespaces.accept('*/xml').get()

Each handler will have a handle method which will be given a client request and be expected to return a client response. Each handler will also be responsible for calling client_request.next_filter to ensure that all filters are chained correctly. For example, the following is the implementation of the handler method in the logging filter shown above.

def handle(self, client_request):
    logger = logging.getLogger(self.log_name)
    try:
        logger.info(client_request.method + ' ' + client_request.url)
    except:
        print 'Error writing request to log'
    client_response = client_request.next_filter(self).handle(client_request)
    try:
        logger.info(client_response.url + ' ' + 
            str(client_response.status)+ ' ' + 
            client_response.reason_phrase)
    except:
        print 'Error writing response to log'
    return client_response

Implementation

I chose to use urllib2, its low level, has a lot of features although not as many as Joe Gregorio's httplib2, but have also a lot of experience using it. I may well change to using httplib2 in the future and there's no reason the Guernsey API should change whichever underlying module I use.

On the name, if you haven't figured it out yet, both Jersey and Guernsey are part of the Channel Islands.

Thursday, November 17, 2011

Minifying CSS - inline images

Having worked on a pretty complex web application, a lot of JavaScript with Dojo, we had to tackle the common problem of browser load performance. Dojo provides some pretty good build tools to minify the JavaScript used by your application, and does compress all the CSS in your application directories as well. The issue is that our CSS references a lot of images, either background, buttons, or other images used in widgets.

So, one possibility is to actually inline the images themselves into your CSS, and with compression and concatenation of the CSS you get to one nice big resource rather than a whole bunch of related and smaller resources.

The trick is to identify all the CSS rules that have values of the form url("...") or url('...'), which are relative file locations. Now, load that resource, convert it into a base-64 encoded value and write this back out as a value of the form:

url('data:image/png;base64,...encoded-value...')

We use the simple Python script below to process a single CSS file, then a shell script that finds and applies this to all our CSS files before we run the standard Dojo build process.

#!/usr/bin/env python

import base64, logging, os, os.path, re, sys

log = None

RE_URL = re.compile("url\([\"'](?P[^\"']+)[\"']")

def init():
    global log
    logging.basicConfig(level=logging.DEBUG)
    log = logging.getLogger('cssmin')

def parse_command_line():
    " returns (options, args) "
    global log
    if len(sys.argv) == 0:
        print "Must specify a CSS file"
        sys.exit(1)
    return ({}, sys.argv[1:])

def encode(file):
    input = open(file)
    encoded = base64.b64encode(input.read())
    input.close()
    return encoded

def process_css(css_file):
    original_file = "%s.original" % css_file
    os.rename(css_file, original_file )
    log.info("Reading file %s." % original_file)
    input = open(original_file, "rt")
    log.info("Writing file %s." % css_file)
    output = open(css_file, "wt")
    for raw_line in input:
        match = RE_URL.search(raw_line)
        if match:
            url = match.group("url")
            src = os.path.join(os.path.split(original_file)[0], url)
            type = os.path.splitext(src)[1]
            if type[1:].lower() in ["gif", "png", "jpeg", "jpg"]:
                data = encode(src)
                output.write(raw_line[:match.start("url")])
                output.write(
                    "data:image/%s;base64,%s" %
                    (type,data)
                    )
                output.write(raw_line[match.end("url"):])
                log.info("inlined resource %s." % src)
            else:
                log.info("ignoring url for type %s." % type)
                output.write(raw_line)
        else:
            output.write(raw_line)

if __name__ == '__main__':
    init()
    (options, args) = parse_command_line()
    process_css(args[0])

Tuesday, October 18, 2011

An Apple users first week with Android


I recently gave up on my aging iPhone 3G, the GPS hardware had failed, the upshot of which was that if I turned on GPS in any app the phone slowed and locked up. The lack of software updates for the older iOS on the 3G meant that worked turned off access to the corporate email because Apple even stopped issuing security patches for older devices. Finally, battery life, even without corporate email, was so poor I could easily get caught out without a usable phone. Also, and most disturbing, my 16Gb phone was full, I'd used it all up on music, I'd stopped synching TV and movie files, then eBooks, but eventually it stopped synching all together.

So, I waited, and waited, would the rumored new iPhone 5 be everything I wanted? Could I wait that long (this was back in August)? Also Amazon announced the Kindle Fire, a simple but nice Android tablet with a great price, I knew I wanted an Android tablet, even though as an Apple household we had an iPad. I really never objected to the Apple lock-down of my phone, after all I always needed it to be able to take a call and didn't want rogue apps turning it into a brick, but with the table, why can't I develop on it? why am I restricted from running an interpreter? I know Apple did lighten up on some of that, and I did run scheme on the iPad which was kind of fun but I coveted the tablets I could shell into and run Python. So, the Fire is definitely on my wish list.

So, October arrives and the iPhone 5 became the iPhone 4S, I've seen colleagues with different Android phones, Droids, Atrias, Galaxys, but certainly the Galaxy S II seemed to have both traction and really looked nice. So, when my 3G finally because just too much of a pain, I bought a Samsung Galaxy S II to replace it, upgrading my iPhone on my AT&T plan which was a bit of a circus but eventually managed to keep my unlimited data. 

So, Thursday, in a meeting a few of us were talking phones and when I said I'd gone from iPhone to Android (or should I say iOS to Android, Apple to Samsung, or iPhone to Galaxy S II?) I got the response "but isn't that a downgrade?". Which caused me to think, what have I given up, what isn't so nice about the Samsung phone or Android OS? The fact I even asked the question implies that I assume that I've given something up, right?

So, after a full week with the phone, and I use my phone a lot so its a short enough time to remember first impressions but long enough to have made some lasting ones too, I wanted to muse on that thought for a while.

Setting up the phone was easy, there were no manuals apart from the usual fold-out getting started guide, but I was able to figure out the general operation and the Androidy way of doing things such as the fixed navigation keys etc. I love the screen on the Galaxy (I'll drop the S II now), and while I know the iPhone 4/4S has a great screen the brightness, depth of color and size (4.3") is really nice. The phone is fast, I mean really fast and with the new AT&T network it browses fast and downloads fast and while its not so fast at home in our cell-challenged area, it actually works which was always at best a 50/50 proposition with the iPhone. So, in initial impressions, did I lose anything? well I guess I lost rounded buttons, the Android buttons are very square, there's no alpha-blended lighting effects either, but funnily enough the buttons work just fine without that - wow, who'd have known? 

Also, in terms of giving things up, I knew I needed to get my music back on my device, or did I? Along with the iPhone 4S launch we heard about iCloud, but to be honest Apple is so late to the party all the good drinks and nibbles were finished ages ago. All my personal email is with Google, I use Google reader, documents on Google Docs and Evernote, and now I have my Amazon CloudDrive and Cloud Player. So do I need that much on-phone storage any more? For this first week I decided not to sync more than I had to, it worked fine, my email, contacts, docs and more connected with my phone easily and seamlessly, I listen to my music and ebooks on the bus streaming from Amazon, read using my Kindle reader and while some of those things wrote to the phone it's really now just a cache, not long-term storage. This really does feel much more like a cloud phone than the iPhone will for a while.

So, I started to think about my next bug-bear for usability, consistency, one of the nice things about the Mac as a platform which we expected to transition to iOS was that Apple's design guidelines and aesthetic have led to a consistency of look and behavior across apps on the platform, which directly leads to the easy adoption of the platform by new users, when you learn the basics most apps seem pretty familiar. Now, I know there are exceptions to this, in fact Apple has some of the most egregious exceptions (can anyone explain the UI on Aperture?) its far more consistent than the alternatives. The interesting thing is that this consistency didn't really transfer to iOS, in many cases you feel that they did try and provide a common way to do things, you imagine Apple designers asking "OK, we need to figure out how users should navigate this feature. So, how would Steve want to do it? ... Great, then well do it that way and the users will just have to behave like Steve.". However, it really didn't follow through, here are two annoying examples.

Settings, so where do I configure my app? some apps like the stocks and weather apps use the little "i" in a circle icon that you click to make changes which was taken from the Mac Dashboard. Some have settings buttons and pages in the app such as Twitter which I used a lot and finally some contributed settings into the "Settings" app. While the first has the advantage of locality, the last has the advantage of consistency, the very existence of three different models is confusing.

Delete, OK, this sounds really anal, but its a good indicator on how the user experience team thinks about common and repeatable tasks. In many apps I need to delete items, so how many gestures can the iOS team come up with to delete things from a list? Well, the Mail app does two things, the stocks and weather apps do something different, the Text app is slightly different and with this guidance apps have come up with even more schemes to do a simple thing like delete.

So, how did Android stack up? Well, first let's re-imagine our design conversation from earlier, "OK, we need to figure out how users should navigate the home page?". For Android you get the feeling the response was "Well the users kind of didn't know, they thought you might swipe from side to side, some said it would be cool if you could tilt the phone to make the icons slide about, others suggested that you could hold down your finger and speed scroll, others liked the on-screen organization but wanted a place to see all their apps alphabetized." . So the response? "Cool, lets do all of them!", So Android can be a little confusing at times when you put your finger on the screen expecting behavior #1 but because you left your finger in contact slightly longer than normal you invoke behavior #2. But I guess it does allow you to pick a style you like, after all with the iPhone if you dont think like Steve the experience can get very frustrating after a while.

So how did the Android do on the Settings and Delete tests? Well on settings it did well, Android clearly separates out two classes of settings, system settings and app settings. Systems settings are accessed via a stand-alone app and pretty much cover what you'd expect, including your accounts which then control email access and so forth. The addition of the dedicated menu button really helps, apps now have a consistent way to present a menu to users, including a settings option. Settings windows/views have a consistent look and feel and are on the most part well layed out. I changed font sizes on a number of app, adjusted sync settings in email and calendars and all those first-week tweaks without any fuss or confusion (except changing the default font on the SMS app which apparently is not possible). 

As for delete, Android is no worse, but no better, Most apps support "press long enough on an item and get a menu, which include delete" behavior which is great, for one-by-one action. The gmail app and others leave a checkbox on the screen next to each item, when you start selecting items you get a delete button (or move or other action), Messages, Mail and others have a Delete menu option that then adds check boxes to the display although Mail has them on the left, Messages and Files put them on the right with a "select all" option.

So, bottom line, did I give up anything? not really, I got a better screen, a much faster phone, I like the ability to manage the home screen(s) with icons and widgets and all my cloud services (and yes, Amazon Video on Demand works just fine so was watching TV on the bus home too). Do I miss the comfort of the iPhone as a device i knew, and its intimate connection to my Mac, well yes, but right now am definitely impressed with both Android and the Galaxy.

Thursday, September 8, 2011

Google Code + Git = Fail

Well, I wanted to start a new project on Google Code and typically have used SVN before but as I'm using Git more and more on my machine for personal work it seemed a great chance to use their new Git support. So, I created a new project, set Git as my version control and went straight to the command line and cloned my new project.

% git clone https://myusername@code.google.com/p/myproject/ 
Cloning into myproject...
warning: You appear to have cloned an empty repository.

Well, that all seemed good, so I added a file, committed locally and tried a push ...

% vi LICENSE
% git add LICENSE 
% git commit
[master (root-commit) 4551a7d] Added file
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 LICENSE
% git push
fatal: https://code.google.com/p/myproject/info/refs not found: did you run git update-server-info on the server?

Oops, so what did that mean? I went to the web page for my project, to the Source tab and browsed the source and clicked on the "git" node; the Filename pane now displays "Error retrieving directory contents." which seems odd, after all there's nothing there but it should at least be able to display an empty project. So a little search suggested that there had been an issue with Google Code and projects that end with a trailing "/" when you cloned them, so I started all over but cloned https://myusername@code.google.com/p/myproject instead. The result? Absolutely the same.

So, I wondered, Google now recommends that you use the .netrc to store your login details and use a URL that doesn't include your email address. So, I added my details into .netrc and tried the whole process again. And this time? Absolutely the same result. So as far as I can tell, from my Mac, using Git 1.7.5.4, I cannot clone and push changes back to Google code :-(

I went to file a ticket and discovered someone else already had so I added some comments and now I guess I wait. You can follow along with the support ticket if you wish.

Monday, July 25, 2011

Add CSS to Dojo Widgets

Many Dojo widgets have custom CSS and you have to remember to include them either in your application HTML pages or in your application CSS file. The problem is that when we add such a widget into our application code we have to figure out whether it has it's own CSS we have to weave in. Usually we find this when we load the page and it looks a mess. So, a widget declares it's templates, it declares the i18n bundles it uses, why doesn't it also include the CSS as well?

What I want to be able to do is something like the following, add a require call to declare my stylesheet and have it dynamically added to the page.


dojo.provide('foo.bar.MyWidget');

dojo.require('dijit._Widget');
dojo.require('dijit._Contained');
dojo.require('dojox.dtl._Templated');

dojo.requireLocalization('foo.bar', 'MyWidget');

dojo_requireStylesheet('foo.bar', 'MyWidget', 'myWidgetClass');

dojo.declare('foo.bar.MyWidget', [dijit._Widget, dijit._Contained, dojox.dtl._Templated], {

    ...

So, I created the dojo.requireStylesheet method shown below. Right
now it's pretty simple, it takes a module name and the base name for the CSS stylesheet so that the result looks much like "requireLocalization". The additional parameter is a class name from the target CSS that we know exists, this allows the method to avoid attempting to load the stylesheet if it has already been loaded (either in code or in the HTML).


dojo.loadedStylesheets = {};
dojo.requireStylesheet = function(/*String*/moduleName, /*String*/styleSheetName, /*String*/expectedClass) {
 // summary:
 //  This call will loaded a CSS stylesheet dynamically, by adding a new
 //  link DOM node to the page head. The caller specifies the name of 
 //  the module owning the CSS, and the call assumes that all CSS files
 //  exist within a "resources" sub directory. The stylesheet name is
 //  again the base name only and the call will append ".css". The 
 //  purpose of this is to allow a requireStylesheet call to look and
 //  feel like existing calls such as dojo.requireLocalization. Note
 //  that best effort will be made to ensure that the same stylesheet
 //  is not loaded more than once and the caller can provide a class
 //  name from the target stylesheet that is expected to be loaded, if
 //  this class is found then it is assumed that the stylesheet has
 //  been loaded elsewhere.
 // moduleName:
 //  the base module name owning the stylesheet. All stylesheets live in
 //  a subdirectory named "resources" of the owning module.
 // styleSheetName:
 //  the base name, without the ".css" suffix, of the stylesheet.
 // expectedClass:
 //  a class in the target stylesheet to be used to test whether it 
 //  has already been loaded. Note that if this is undefined the call
 //  will always try and and load the stylesheet.
 //
 function onClassFound(classItem) {
  if (classItem === null || classItem === undefined) {
   var url = dojo.moduleUrl(moduleName + '.resources', styleSheetName + '.css');  
   var link = dojo.create('link', {
     type: 'text/css',
     rel:  'stylesheet',
     href: url},
        dojo.doc.getElementsByTagName('head')[0]);
   dojo_loadedStylesheets[styleSheetName] = true;
  }
 }
 if (moduleName !== undefined && styleSheetName !== undefined) {
  if (dojo_loadedStylesheets[styleSheetName] === undefined) {
   if (expectedClass !== undefined) {
    var classStore = new dojox.data.CssClassStore();
    classStore.fetchItemByIdentity({identifier: expectedClass, onItem: onClassFound});
   } else {
    onClassFound(undefined); /* force loading */
   }
  }
 }
}

Now your widgets have the following structure, and while a lot of Dojo widgets and user widgets use the "resources" directory the code above makes that a requirement in the same way that dojo.i18n requires the "nls" directory.


/ foo.bar
  +-- nls
      +-- Mywidget.js
  +-- resources
      +-- Mywidget.css
  +-- templates
      +-- Mywidget.html
  +-- Mywidget.js

TODO: one issue with this is that it is not "theme" aware, the issue today is that the notion of theme is encoded in the CSS (and image, and other) resource URLs. A common approach to the identification of the current theme and encoding the theme in the path would be really valuable - if anyone from Dojo is listening :-)