Fast Prototyping

Every once in a while, there is a need to create a working prototype of some experimental functionality. Lets say you are a programmer. A marketing guy asks you to make something up from a wireframe he gives you... Of course this is wrong right from the beginning - in a large project you should consult the Architect before making any add-ons or changes to the application, Product Management shouldn't report into marketing, and he probably forwarded it straight from the customer anyway... Sometimes you need to try something out. Show it to your customer and your Q&A team, tell them it's an alpha. Show it to your users and tell'em it's a beta. Then your marketing boss comes along and says: "Good job! It's working! Leave it!". At this point, the new product made it through a release cycle, but users didn't start using it. After a while, it stops working...

So what is a prototype? Some people may have an image of libraries or other components stuck together to appear like a finished product (which it's not). A prototype lacks key elements of software design and implementation, like modularity, documentation, unit tests... It can have unpredictable capacity limitations (because of limited testing and no design phase). The resulting code will be unreadable to developers who didn't participate in it's creation.

If you tell a programmer to make it quick, or hurry up, he will make something more or less fitting this description.

My experience suggests that if you continue on making iterative changes to the prototype, bad things happen:

1. Your new project can suddenly become just too big. Even if you use frameworks with well defined design patterns, like MVC, no one person can grasp all relations between modules, changes in state and logic, etc. (no technical leadership).
2. If you didn't do documentation, there is a small number of programmers that understand and work on the core application (which usually is just a fragment of the first prototype).
3. Some functionality can become very difficult to maintain after an author leaves the workshop.
4. Poorly designed or not documented code accumulates bloat, becomes slow and buggy.
5. If you forget about tests, you can't easily rewrite different modules inside the core without making changes in the whole package, changing APIs and introducing bugs. Your current API is opaque and hard to debug.
6. If you didn't document all application features, you can't simply rewrite it, in fact, it can be quite impossible.

Now you have two options - dump your current customers (or wait for them to fire you), or sell your users/clients a new, rewritten version of the product, with new functionality and better usability. I've seen it go both ways.

I won't look further into the problem of project management. I'm more interested in what can we do by changing the solution's architecture.

When making a prototype, an average programmer, will dump his blob of code, thus creating his application; a resourceful programmer will use many external calls to libraries, toolkits and frameworks which the average programmer hasn't heard of, writing less code and finishing his assignment earlier; but a successful programmer will also create tools of his own. The premise here is that libraries have an interface, tests and documentation.

Using libraries, means less code to develop and less bugs. Frameworks provide a solid, well tested base, free built-in functionality, examples and common programming style. Many tools are also component based and have a consistent terminology - these are pretty important in a large project with many developers. The downside is always the learning curve, sometimes performance.

Am I against prototyping? Surprisingly, no! Even if you are average, or have average programmers on your team, you can plan to fail, and plan rewrites. This is what good Product Managers do (they might be hiding this fact from you).

Some time ago, I read an opinion, that stated you have to (re)write your application three times, because the third version is generally the best, it goes something like this:

The first is the "I don't know what I'm doing" version, which gets written by trying without much thought, ugly hacks, and without a decent design. Sometimes it does work quite well however, as despite not being very pretty it does what it's supposed to.

The second is "V1 is crap, but now that I have figured it all out I can do better!". Often a horrible mess, due to things like wanting to make everything modular, adding every feature possible, and using the latest cool tech and design patterns where they don't belong. Turns out to be slow, huge, buggy and overly complicated to use. There's even a name for this: "second system effect".

Based on the lessons learned from the lack of planning in the first and the excesses of the second, the third version has a good chance of being actually decent.

[source]

However careful is your planning, you may still want to be on the edge of innovation, and release unfinished software (and be more like Google). Maybe you want to start an Open Source project, which should gain momentum, not otherwise.

I will explore my ideas on how to plan for big and small changes in my next post.

As a sidenote: if you are new at this, and you just found out your boss is from marketing, don't do prototyping, forget about quick hacks when bug-fixing. If you plan on keeping your job longer than a year, focus on writing good code, learn what the current application does. Your work will be appreciated by your coworkers. If you see others do it, stop them, or start looking for a new workplace. Be careful not to become a superstar developer.

Infinitely configurable web application

Today, I wan't to entertain the idea of making an application, which behavior can be modified in any way by the user (by user, I mean the administrator of the server running your app).

Most applications have a notion of a config file, which usually is just a collection of setting-configuration pairs. When a project gets bigger, settings are separated into sections. If the problem domain requires it, sub-sections can represent similar entities, with the same available settings, but different configuration. Think about a web server, which can serve multiple domains, configured in various ways.
Sometimes the sub-sections can create a never ending tree. The configuration is then being derived from the root node and all branches on the path to the node, which configuration we want to retrieve. Apache .htaccess configuration files come to mind.

How does this map onto web applications?

A web application has a tree structure, from the top, there are:

1. domains,
2. URLs,
3. pages,
4. parts of a page,
5. elements on a page.

Getting right to the point, an example configuration file for this blog could look something like this:

<Site domain="blog.xek.pl">
  <StaticPage url="/">
    <SpanVertical>
      <Header title="XEK"/>
      <SpanHorizontal>
        <LastPosts no="7">
          <PostTitle with_date_header="true"/>
          <PostBody/>
          <SpanHorizontal>
            <PostAuthor/>
            <CommentsLink/>
          </SpanHorizontal>
          <PostLinksTo/>
          <PostLabels/>
        </LastPosts>
        <SpanVertical width="10">
          <BlogArchiveLinks/>
          <AboutMeSection>
            <Avatar/>
            <Name description="true"/>
            <ProfileLink text="View my complete profile"/>
          </AboutMeSection>
          ...
        </SpanVertical>
      </SpanHorizontal>
      <Footer/>
    </SpanVertical>
  </StaticPage>
  <ObjectPreviewPage url="/[object_id].html" object="BlogPost">
    <SpanVertical>
      <Header title="XEK"/>
      <SpanHorizontal>
        <BlogPost>
          <PostTitle with_date_header="true"/>
          <PostBody/>
          <PostLabels/>
          <PostLinksTo/>
          <PostComments/>
          <PostCommentForm/>
        </BlogPost>
        <SpanVertical width="10">
          <BlogArchiveLinks/>
          <AboutMeSection>
            <Avatar/>
            <Name description="true"/>
            <ProfileLink text="View my complete profile"/>
          </AboutMeSection>
          ...
        </SpanVertical>
      </SpanHorizontal>
      <Footer/>
    </SpanVertical>
  </ObjectPreviewPage>
</Site>

You don't have to make everything configurable at once. In first iteration you can make your app accept different settings depending on which domain (or installation) it is run, but shortly you will notice, that many pages are similar and belong to one type or class, you also have many repeating and common elements on pages.

Implementation details

I'm sure you noticed the use of CamelCase in my tag names, this is because everything from a Site to a CommentsLink can be implemented as a class. The class is then initialized with the configuration file, making the above file an instance of the Site class. The only thing you need to remember is that a child element in this configuration tree, must implement all interfaces the parent element want's to call. In the above example the BlogPost and LastPosts call the same API, and as a result, all children of these elements can be freely swapped between them. The above example shows only the data presentation part, but you can do editing, forms and fields just as easily. If you are implementing tabs or steps with different URLs, you may want to make the Page objects able to call themselves. As a side-note, you probably want to make the number of these interfaces small, so more elements could be used in more places. Interfaces should be documented.

How does this compare to current web-frameworks?

Well first of all, when I think about web-development frameworks, I think about Django. That being said, the above configuration scheme, shouldn't be to hard to implement. Modifying the core framework won't be necessary - this is because all configuration files are Python executable code.
The main problem in Django, is that all Controllers (called Views) are just functions. Their configuration is partially stored in urls.py, which specifies the URL path and potentially other configuration (which is passed as function arguments), and partially in settings.py, which usually just stores setting-configuration pairs. Having a function, you don't know what interface it implements and what other APIs are called internally, making the Views themselves hard to unit-test. You can stack function calls, but then, configuration becomes embedded inside your code. I know other frameworks implement Controllers as classes, which is a big step forward, making Convention over Configuration do wonders. The problem with these classes is, that they operate as singletons, what in my opinion, is a big needless constraint. In contrast, the "Infinitely configurable" design pattern instantiates each class with a configuration, making it a reusable component in more than one place of your application. There are other constraints in web frameworks, which you have to take into consideration - I will only mention Django template-tags and forms, each of which enforces a set of unchangeable rules (you have to edit templates to move a template-tag, you can't just move an edit form implementation to another arbitrary page).

Next step

Up until now, I talked about advanced users, users which can read and edit an XML file, but lets take it to the next level. You probably noticed the above file is kind of repetitive. This stops to be a problem when you move it to a different data structure, inside a database! Now, if you have it inside a database, you can make a nice admin panel for it :) . Each class, can render it's configuration form, and all classes can export information about where they can be placed in the configuration structure, in respect to other elements. This makes the application very, and I mean Very configurable. I leave the rest to your imagination.

What are the benefits?

Your application is now a product. You can distribute it to end users and they can at least disable features they don't want. If you implement it right, the code for these features won't be even loaded when the application is started, so your app can stay competitive to any newcomers.

Less code. After a certain break point, you may find out that many of the new features requested by your users, can be implemented only by changing the configuration.

DRY, by example. Let's assume you have a standard API, for editing forms, it is implemented in a class, from which other functionality inherits. As your site becomes more AJAX-friendly, a client requests a registration box, a newsletter box and also to move the poll to an AJAX reloaded page element. In my configuration scheme, you would write an AjaxWrapper class, and then change configuration for this one client.

Let me know what you think.

Hello World

Welcome to my new blog, on which I grok architectural and low level problems in web application development, my goals being: quick but extensible implementations, readable and easy to review code, efficient and predictably scalable solutions.

About me

I have a 5 year experience in web application development - programming in Python, JavaScript and Erlang, and some other previous experience (like my abandoned mobile jabber communicator and my SoC project). I worked for 2 years at grono.net, which, at the time, was the largest social networking portal in Poland, and the largest deployment of Django in the world. I later moved on to work at Sensi Soft, which became The Python Development Shop in Warsaw. I had the chance of working and providing solutions for clients all over the world: Harte-Hanks "North America's largest owner, operator, and distributor of weekly shopper publications", Friday-ad "the largest Independent publisher in the UK", Junk Mail "Your King of Classifieds" from South Africa, Onet.pl "the biggest and most popular multimedia portal in Poland" (I can confirm this one), itself a part of ITI Group which owns TVN, a leading Polish free-to-air television channel. At Sensi Soft, I worked in a position of a Team Leader (together with more then 20 developers in one project), and later became the Solutions Architect.