Instant XML feeds via the JSTL SQL tags

20 December 2007 » DB2, Java, MySQL, Web architecture, WebSphere, XML

A dusty old Java tag library can help conjure up siloed Web site data for new uses.

Some background
I’ve developed a number of server-side Java Web applications over the years, first with scriplets embedded in JSP, then with the template and tag driven paradigm offered by ATG Dynamo before the J2EE standards, and most recently with the Model-View-Controller architecture pattern in Struts and Spring MVC.

Each of those technologies (mostly) improved on its predecessor and enforced a better separation of concerns between the database, application logic, and presentation of the end result in the browser. This in turn has helped my teams divide and conquer Web application development among specialized job roles.

That’s why I’ve long been puzzled why the SQL tags in the JavaServer Pages Standard Tag Library exist as a standard part of J2EE 1.3 onward. These tags enable a front-end developer to embed SQL directly into a JSP page without the need for scriptlet code.

This tag library seemed an ill-conceived reversion (anti-pattern even) to the days before MVC took hold as a best practice in the Java world, and I’m pretty sure I skipped that section of the objectives when studying for the SCWCD exam.

That said, the SQL tags came in pretty handy this week for a particular challenge, and the more I think about how I can use them beyond their intended purpose, the more every new requirement I see looks like a nail.

My particular application context
I support a content management application which was designed, developed, and deployed circa Web 1.9. It’s stable, performant, and most importantly, met its functional requirements of the day.

In the two years that it’s been deployed, several new requirements have arisen that have expanded its anticipated scope as a traditional Web application.

In particular, the ubiquity of XML feeds have driven the need for it to present its core data outside of the templates existing in the confines of its own Web site. The rise of tagging and the popularity of multimedia as syndicatable content has also made it creak.

Compounding the architectural limitations of the application itself is its inflexible hosting environment. The data center that this site is deployed to is governed by CYA-driven restrictions (rightly so) which constitute a barrier to frequent application deployment cycles that add new functionality.

This environment makes it difficult to adopt nascent technological advances – the next big thing in “coolness” or usability – but have also kept it exceptionally stable and available to meet its codified requirements without introducing undue legal or financial risk.

The application itself consists of two subcomponents. There is a Web application module on the secured intranet for authors to generate new content, and a publicly accessible read-only Web application module to display published content.

It’s primarily this latter Internet application where the use of JSTL SQL tags comes in most handy, but I can imagine uses on the intranet side as well (ad hoc reports, for example).

The case for SQL tag driven XML feeds
The JSTL standard defines a tag library for issuing queries against a data source defined in the Web deployment descriptor without using JDBC in Java scriptlet code in a JSP.

If this sounds like a simple concept that harks back to the type 1 JSP days, that’s because it is. The documentation shows its own apprehension about the inappropriate use of these tags:

The JSTL SQL tags for accessing databases … are designed for quick prototyping and simple applications. For production applications, database operations are normally encapsulated in JavaBeans components.

But therein lies their simplicity, flexibility and power for this particular production application scenario.

In order to take any slice of your data that can be exposed via a SQL query to the authorized user mapped to the JNDI entry for that data source, all you need to do is is write your query and iterate through the result set in an XML template defined in your JSP.

Think about that outside of this technology’s intended use as a prototype or simple application building block. Instead, imagine how you could use these tags to improve the value of a complex existing production application.

For example, suppose you’ve always provided an RSS feed for your latest ten published news stories. You’ve written your Controller or Action in your chosen MVC framework of choice and deployed it.

But now your users are demanding the latest five thumbnails of images published with a story to accompany its syndicated title and abstract in their latest mashup. Or perhaps they only want to see the last 10 stories which contain a given keyword.

What do you do? You could write a new Action or Controller and proper Command class in Java to meet that requirement. That would require updating some configuration files or deploying an EAR or WAR.

But look, you have an existing deployed stable application. Why risk introducing new code or downtime to a perfectly good application? Why not just free your data for use by your users’ new requirements in a quick hitting, low risk way?

Reuse your data by plugging in new JSTL SQL tag driven JSP files, don’t rebuild your application for every new data usage requirement.

To the tag library!
Ok, so you’ve read this far. I promise, the implementation itself will be much shorter :)

So your users want more information delivered via your feeds, or they wish to query by keyword or otherwise filter your data in a way you never anticipated.

Let’s see if we can free up that data for them.

  1. Write your query, with or without input parameters.
    [code lang=”sql”]
    SELECT ID, TITLE, ABSTRACT FROM NEWS_ARTICLES;

    SELECT ID, TITLE, ABSTRACT
    FROM NEWS_ARTICLES
    WHERE BODY LIKE ‘%?%’
    FETCH FIRST 5 ROWS ONLY;

    SELECT ID, TITLE, ABSTRACT, THUMBNAIL
    FROM NEWS_ARTICLES NA, NEWS_ARTICLE_IMAGES NAI
    WHERE NA.ID = NAI.NA_ID;
    [/code]

  2. Determine what XML format it should be in, whether a standard such as Atom or something custom like the following.
    [code lang=”xml”]







    [/code]
  3. Tie the query to the format in a JSP file using the JSTL SQL tag library (and optionally, the Core tag library to escape output) and the JNDI name of the data source you already have configured in web.xml.

    Consult the documentation if you want to use placeholders.
    [code lang=”xml”]
    <%@ page contentType="text/xml; charset=UTF-8" pageEncoding="UTF-8" session="false"%>

    <%@ taglib prefix="sql" uri="http://java.sun.com/jsp/jstl/sql" %>
    <%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %>



    SELECT ID id, TITLE title, ABSTRACT abstract, BODY body, THUMBNAIL thumbnail
    FROM NEWS_ARTICLES NA, NEWS_ARTICLE_IMAGES NAI
    WHERE NA.ID = NAI.NA_ID;



    “>
    <c:out value="${row.title}"/>




    [/code]

  4. Deploy the JSP file as your application server requires. If reloading is not enabled, restart the application (consider setting a 15 minute timeout or similar, so you gain the performance boost but provide a hook for updating JSPs individually).

Conclusion
That’s it, you’ve now added an aspect of functionality to your application which frees up any data you can query for via SQL (or XQuery, if your data server is so enabled). You’ve done it in a pluggable fashion and haven’t needed to build any new Java code within your existing application and its framework.

Of course, the flip side is that you’ve done it outside of your application framework and may have circumvented some well-intended best practices. However, you may to prefer to think of this approach as a temporary, low-risk way to share the data available to the users of your application in novel ways that may justify investing in the development of longer term solutions.

Ironically enough, this reversion to single file deployment can make an application buzzword compliant with the one of the most touted recent enterprise targeted architectural pattern – SOA. It reduces the barrier between the value an application has – its data – and the consuming end point of that data – to a simple JSP.

All married up and honeymooned out

17 December 2007 » Potpourri, Travel

Ah, so it’s been close to two months since I last posted, but I have a good excuse.

Cat and I were married on November 24th, the Saturday after Thanksgiving in Tarrytown, New York. We were thrilled that so many of our friends and family could travel so far or otherwise be there on the holiday weekend. It meant a lot to us.

Our excellent photographers Justin and Mary have posted a few teasers up on their blog – we can’t wait to see how the rest came out. This shot in particular sums up both their creativity and their drive to know their clients ahead of the big day. We’d recommend them without a moment’s hesitation.

Our mutual friend and former colleague Jen König posted a pair of beautiful pics on Flickr as well; Cat’s mom lighting the unity candle during the ceremony and our entrance at the reception.

Speaking of Flickr, I finally broke down and signed up for an account to post close to 600 of the “best” shots from our honeymoon in New Zealand. It’s hard to believe that even that many is a mere quarter of the 2112 shots we took :)

Thoughts on the Spring Framework

One of my goals for the year was to learn more about the Spring Framework, a layered set of modules that intend to make enterprise Java development easier.

In September, I took a class on Spring. Last month, I put together a presentation to share what I learned with my colleagues.

The bootcamp
I attended the Core Spring bootcamp in New York City. If you’re looking to learn about Spring in a classroom environment, there is perhaps no better source than Interface21‘s consultants as they are the company behind the project.

The course materials were solid and the class revolved around a real-world application scenario. For each segment presented, corresponding before and after projects for the lab were available in a workspace used by the free, Eclipse-based Spring IDE that was distributed to the class.

A copy of Pro Spring was included with tuition, but I’ve found the newer Spring in Action, second edition a better read. The instructors also noted that the older Pro Spring book had information on AOP and transactions that weren’t up to speed with the latest 2.x versions of Spring.

As you would expect from a technology that in practice killed EJBs, the Spring bootcamp focuses on middle-tier applications. However, there was good coverage of Spring MVC, which I hope to a adopt for a new project built on WebSphere 6.1 and J2SE 5.0.

Like Baldwin home from Paris
Some time after the course I had the opportunity to present an overview of the Spring Framework to my department. The audience consisted of developers as well as project managers and other non-technical types, so the goal was to keep the presentation accessible as well as sufficiently informative.

To sum up the philosophy of Spring in a nutshell, I hit on three characteristics with real world parallels. I hope they went over well, and I think some of the other learning materials out there would benefit from a similar approach. It seems the ideas behind Spring could be conveyed easier than they often are (another reason I like Spring in Action).

  • Inversion of Control / Dependency Injection
    Give code what it needs, don’t make it ask
    Does your car buy its own gas? Should it have its own credit card? Or use yours?
  • Interchangeability through Interfaces
    Keep things “black box” enough that you can swap individual pieces out when you need to
    Your twenty year old lamp can take either a traditional incandescent light bulb or a new compact fluorescent light bulb which provides a 75% more efficient way to do the same thing in the same standard socket.
  • Aspect Oriented Programming
    Terrible name and difficult vocabulary for an easy-to-grasp concept
    Consolidate logging and access control code, don’t repeat it all over the place
    Imagine an assistant tracking your billable hours to several projects during the day for you while you focus on your real job.

Silly middle-tier guys, or, what’s with that goofy .htm extension?
I have one outstanding gripe about Spring. Spring MVC in particular. I let it slide in the class, but since the foolishness is repeated in other materials for learning Spring I felt I had to speak my mind.

In a lot of examples for configuration of the Spring MVC DispatcherServlet, folks will tell you that “.htm” is the preferred convention for the mapping requests to the front controller.

The argument goes that we should trick the search engines into thinking this is static HTML content and besides, the mapping reflects that even though the page is dynamic, it is indeed rendered in HTML. Furthermore, the “.do” convention from Struts is pure tomfoolery.

This is dogsqueeze.

I take exception with all those arguments. In fact, I’d go as far as to say the “.htm” extension for dynamic content is actually harmful:

  • Most developers deploy Java Web applications to an application server which is not also the front-end Web server. Application servers, such as WebSphere, are configured to serve only dynamic content, such as servlets and JSPs. The HTTP server, such as Apache, serves the static files such as HTML documents, stylesheets and images.

    What if you really did have a static file with a “.htm” extension deployed by one of your Web developers? Do you want to see her go bonkers when she can’t figure out why her file is not being found because you decided to map that extension to your servlet? WTF?

  • Do you really want to make the impression that you’re hosting your Fortune 500 or high volume e-commerce site on MS-DOS?
  • The “search-engine friendly” argument for the “.htm” extension goes out the window when you start using query string parameters anyway.
  • You’ve introduced goofiness at the cost of elegance, another tenet core to Spring. Sigh.

But honestly, if you’re going to go through with it anyway, just add the damn extra “l”, filename limitations (virtual as they may be) are a thing of the past.

New Zealand honeymoon planning 2.0

17 October 2007 » Travel

Cat and I have been finalizing our agenda for the post-wedding trip to New Zealand. We’ll be spending two weeks there in late November through early December – early summer in the southern hemisphere.

Update: We’re back from the trip. This was an ambitious agenda for two weeks, and we did change up some things before we left… like leaving out about half of what we planned on doing below…

Here are the pictures from what we did get to see. What a great trip. Highly recommended. :)

We’re not thrill seekers (Cat still gives me grief about bungee jumping 5 years before we met), so we intend to spend most of our time on the North Island relaxing on the beach or hitting the wineries and chowing on local seafood.

Most of the recommendations we’ve gotten from friends and colleagues have been for the more adventurous lower South Island, unfortunately.

So I’m looking to tap the wisdom of crowds (well the 4 of you that regularly read this blog anyway) for North Island and/or northern South Island tips. That’s where the 2.0 comes in (no, neither of us has been married before, thank you very much).

Here’s our the preliminary agenda. Plus ones? Vote downs? Things we’ve overlooked completely?

Day 1: Thursday

  • Arrive in Auckland early morning
  • Sky Tower
  • Auckland Domain
  • Parnell Village
  • Dinner options:
    • Dine by Peter Gordon (in Sky City)
    • White
    • The French Cafe
  • Stay in Auckland

Day 2: Friday

Day 3: Saturday

Day 4: Sunday

Day 5: Monday

  • Kiwi 360
  • Lunch in Bay of Plenty
  • Stay in Rotorua

Day 6: Tuesday

Day 7: Wednesday

Day 8: Thursday

Day 9: Friday

  • Shopping in Wellington morning
  • Take Ferry to Picton
  • Dinner at: ?
  • Stay in Blenheim

Day 10: Saturday

  • Wine tour day
  • Dinner options:
    • Herzog
    • Hunter’s Vineyard
  • Stay in Blenheim

Day 11: Sunday

  • Drive south
  • Lunch in Kaikoura
    • White Morph Restaurant
  • Dinner options:
    • Indochine
    • 50 on Park
  • Stay in Christchurch

Day 12: Monday

Day 13: Tuesday

Day 14: Wednesday

Day 15: Thursday

  • Drive to Auckland
  • Dinner at: ?
  • Depart from Auckland in the evening

Thanks in advance for any tips. Folks with great pointers get a nice postcard. :)

Native XML Databases at NYPHP next week

17 October 2007 » DB2, Java, MySQL, PHP, XML

Elliotte Rusty Harold will offer his take on Native XML Databases at New York PHP next Tuesday night in Manhattan.

The presentation follows a mailing list thread and resulting blog post that generated a lot of interest and discussion on the topic. It should be a great talk for database administrators, application developers and content producers alike:

While much data and many applications fit very neatly into tables, even more data doesn’t. Books, encyclopedias, web pages, legal briefs, poetry, and more is not practically normalizable. SQL will continue to rule supreme for accounting, human resources, taxes, inventory management, banking, and other traditional systems where it’s done well for the last twenty years.

However, many other applications in fields like publishing have not even had a database backend. It’s not that they didn’t need one. It’s just that the databases of the day couldn’t handle their needs, so content was simply stored in Word files in a file system. These applications are going to be revolutionized by XQuery and XML.

If you’re working in publishing, including web publishing, you owe it to yourself to take a serious look at the available XML databases. This high-level talk explains what XML databases are good for and when you might choose one over a more traditional solution. You’ll learn about the different options in both open and closed source XML databases including pure XML, hybrid relational-XML, and other models.

As always, the meeting at IBM is free and open to the public, but you must submit your RSVP by 6PM EDT Monday, October 22nd.

DB2 for Intel Mac

24 September 2007 » DB2, IBM, Java, Mac, PHP

Antonio Cangiano has offered tantalizing news about the upcoming release of a developer’s edition of DB2 Express-C for Intel Macs.

According to Cangiano – a software engineer at the IBM Toronto Software Lab – a beta of the full data server, not just an application development client or driver, should be out by the end of the year.

The interest of Python and Ruby developers helped drive the case at IBM for a Mac version of DB2, but I imagine PHP and Java programmers on this platform are looking forward to the official announcement as well.

I’m still running on PowerPC and it would be nice to see DB2 released on an IBM processor built for Apple, but it’s another good reason to pick up a new Mac around Christmas :)

Elliotte Rusty Harold on native XML data servers

16 August 2007 » DB2, IBM, MySQL, PHP, Web development

Soon after a New York PHP mailing list exchange debating the merits of storing information in hierarchical XML format versus traditional relational tables, XML guru Elliotte Rusty Harold posted a summary of the State of Native XML Databases to his blog.

Like the thread that inspired it, the post has generated a lot of comments showing that it’s an emerging technology whose potential is not well understood and that the products which implement the technology aren’t well known.

Why use an XML database?
Before considering an investment in a data server which offers native XML storage (one which doesn’t decompose it, nor store it as an unstructured chunk; and which allows the user to query its arbitrary individual elements), it’s necessary to take a step back and see what XML as a storage method offers the Web developer.

  • What sort of information should be stored as XML?
    The examples cited by Elliotte include large documents where the document itself is composed of related data, yet which it would be inefficient to break down into related tables and columns. A book can be broken down logically into a title and an abstract but what about the individual paragraphs in each chapter? What of the table of contents and index which are derived dynamically from data which exists elsewhere in the document?
  • Why can’t this data be stored in another format?
    It can be stored, but how do you make use of it? You might shred it, but this requires time to decompose and then recompose, assuming you can get back the data in the form you require. For example, what if you needed the first paragraph and figure of every chapter to compose a detailed table of contents? How would you write that query? What would you do if you needed to add, remove, or reorder a paragraph in an encyclopedia?
  • Why is data stored in XML format increasingly becoming valuable?
    According to Elliotte Rusty Harold and Anant Jhingran, most existing data isn’t traditional relational data at all. There is a ton of information that can not be queried currently with traditional SQL nor stored efficiently in relational tables. Think about the Web itself, it’s a collection of documents that have individual (ideally semantic) structure.

Sound bites
Here are a few of the insightful nuggets which sum up Elliotte’s point of view. For some of his thoughts on the future of XML in general, have a look at his Ten predictions for XML in 2007.

From http://lists.nyphp.org/pipermail/talk/2007-August/022724.html

Roughly 80% of the world’s data cannot plausibly be stored in a
relational database. The 20% that does fit there is important enough
that we’ve spent the last 20 years stuffing it into relational databases
and doing interesting things with it. I’m still doing a lot of that.

But there’s a lot more data out there that doesn’t look like tables than
does. Much of this data fits very nicely in a native XML database like
Mark Logic or eXist. There’s also data that has some tabular parts and
some non-tabular parts. This may work well in a hybrid XML-relational
database like DB2 9.

If your only place to put pegs is a table with square holes, then you’re
going to try pound every peg you find into a square hole. However, some
of us have noticed that a lot of the pegs we encounter aren’t shaped
like squares, and sometimes we need to buy a different table with
different shaped holes. :-)

Relational databases didn’t take the world by storm overnight. XML
databases won’t either. But they will be adopted because they do let
people solve problems they have today that they cannot solve with any
other tools.

From http://lists.nyphp.org/pipermail/talk/2007-August/022788.html

XML is not a file format. We’ve been down this road before. A native XML
database is no more based on a file format than MySQL is based on tab
delimited text.

From http://lists.nyphp.org/pipermail/talk/2007-August/022789.html

Storing books, web pages, and the like in a relational database has only
two basic approaches: make it a blob or cut it into tiny little pieces.
The first eliminates search capabilities; the second performs like a dog.

Also from http://lists.nyphp.org/pipermail/talk/2007-August/022788.html

>> I’m glad we have multiple tools to bring to bear on this kind of
>> problem, because I worry about the performance implications of
>> querying an XML database for the average price of those books, or
>> performing an operation that adds another field (tag?) to each book’s
>> “record”.

Average prices, or adding a field, can be done pretty fast. I don’t know
if it’s as fast as oracle or MySQL. I don’t much care. Sales systems are
exactly the sort of apps that relational databases fit well. But
actually publishing the books? That’s a very different story.

>> If it’s not too much trouble, could you give us some other use cases
>> for an XML database? Because title and first paragraph, if that’s
>> something a system “routinely does” could easily be stored as
>> relational data at the time of import.

Just surf around Safari sometime. Think about what it’s doing. Then try
to imagine doing that on top of a relational database.

Think about combining individual chapters, sections, and even smaller
divisions to make new on-off books like Safari U does. Consider the
generation of tables of contents and indexes for these books.

Closer to home, think about a blogging system or a content management
system. Now imagine what you could do if the page structure were
actually queryable, and not just an opaque blob in MySQL somewhere.

And the takeaway from the State of Native XML Databases:

If you’re working in publishing, including web publishing, you owe it to yourself to take a serious look at the available XML databases. If they already meet your needs, use them. If not, check back again again in a year or two when there’ll be more and better choices.

The relational revolution didn’t happen overnight, and the XQuery revolution isn’t going to happen overnight either. However it will happen because for many applications the benefits are just too compelling to ignore.

Conclusion
This is interesting stuff, and I’m glad Elliotte was able to put forward some of the reasons one might use an XML database and describe the maturity level of the data server products out there now.
FLWOR
We’ve asked Elliotte to present at one of the upcoming New York PHP meetings in October or November. If he can’t make it, it would be interesting to hear from other folks doing PHP work with XML databases, such as the XML Content Store / Zend_Db_Xml in the Zend Framework.

A new tonneau cover for the Frontier

14 August 2007 » Photos, Potpourri, The truck

This past weekend I installed a hard tonneau cover on my Nissan Frontier pick-up truck. I had been meaning to do it for quite some time, but choosing the right cover has been a challenge.

Nissan sells both a factory hard tonneau and a soft cover tonneau but neither is cheap nor easy to install and remove. The many independent hardware vendors sell hard and soft covers which never quite fit my needs.

After several fits of research over the years, I found a company called Lazer Lite that makes a nice hard aluminum cover which opens with the help of a pair of hydraulic struts. It also removes easily and doesn’t take up much room in the bed itself.

Cat ordered one for me as a surprise for my birthday, and although it took a while for Lazer Lite to build and ship, I’m pretty damn happy with the cover and the customer service we received.

If you’re in the market for a tonneau, I highly recommend Scott and the folks at Lazer Lite.

Here’s some before and after pics of the cover on my truck.

Next page »