Skip to main content

Data and Databases ~ Concise and Simple

(As with all "concise and simple" articles I assume no prior knowledge of the subject and keep the length to less than 1500 words.)

"Scientia potentia est." (Knowledge is power.)

Sir Francis Bacon.

In the digital world knowledge is stored, in its latent form, as data in databases. Everyone from your local supermarket to the government collects data about all of us. Information and knowledge is extracted about all aspects of our lives, from our shopping habits to whether we've paid our taxes (it is no accident that the examples above relate to extracting money from us). According to Google it currently takes just two days for us to generate as much information as was created by all of humanity from the dawn of time until 2003.

This article is about data and databases - the technology that makes the accumulation and analysis of this flood of information possible. It starts with a simple question: what is data?

"Data" are values that convey meaningful information about something. Meaning is implied by the context and structure of the data. Structure is formed from the way data is represented. Databases simply store and provide mechanisms for working with structured data.

The most fundamental data-related act is to name a value:

my_name = "Nicholas Tollervey"

My name is Nicholas Tollervey. This is shown by "my_name" being assigned the value "Nicholas Tollervey". Assignment happens when a token name is followed by an equals sign that is, in turn, followed by a value. Notice the three layers: meaning, structure and representation.

Another example:

nicholas = {
    "first_name": "Nicholas",
    "middle_name": "Harry",
    "surname": "Tollervey",
    "birthday": {
        "year": 1973,
        "month": 8,
        "day": 19
    "height": 180,
    "weight": 76.9

After reading the above I'm certain you'll know when to send me a birthday card. Something called nicholas has a single value associated with it: an associative array. Associative arrays are key / value pairs - like a dictionary of words or a subject index at the back of a book. In this case it has six entries, one of which is itself a nested associative array with three entries. It's an associative array because it is enclosed by curly braces and contains a list of "key_name": value pairs separated by commas. The associative array is assigned to nicholas in exactly the same way as the first example (token name followed by equals followed by value).

Incidentally, I can retrieve my data like this:

>>> my_name
"Nicholas Tollervey"
>>> nicholas["height"]

(This example uses the Python programming language: I type in commands after the >>> prompt and the Python interpreter replies on the new line.)

A final example:

id first_name middle_name surname nationality_id
1 Nicholas Harry Tollervey 2
2 Terry Charles Oliver Jones 1
3 Esteve Fernandez 3
id name
1 Australian
2 British
3 Spanish

Obviously, I'm British and my buddies Terry and Esteve are Australian and Spanish (respectively). You know this because you understand how to read tabular information and spot the relationship between the values in the nationality_id column of the People table and the values in the id column of the Nationalities table. You know it's tabular information because it's a two dimensional grid with the vertical columns indicating named fields and horizontal rows representing individual records. Once again: meaning, structure, representation.

Each example is more complex than the one preceding it. The first is a single key / value pair (where my_name is the key and "Nicholas Tollervey" is the value), the second is a key (nicholas) associated with a value that is itself key / value pairs (the associative array). Finally, the third example contains rows of items that all share the same keys (a.k.a. columns) but different values. Each type of record (people, nationalities etc) has its own table.

How do such meaningful data structures / representations relate to databases?

There are four basic actions that all databases may perform on data: create, read, update and delete (often expressed as CRUD). There are many types of database that meet this minimum functionality. The three data examples illustrate the types of data structure stored in three of the most popular types of database: key / value stores (such as Redis or Memcached), document oriented databases (like MongoDB or CouchDB) and relational databases (for example, Postgres or MySQL).

Key / value databases are, at heart, associative arrays. They provide three basic actions:

  • Assign a value to a key (both creating and updating).
  • Read a value associated with a known key.
  • Delete a value using a known key.

Such simple databases are extremely fast and well suited for tasks where speed is of the essence. However, there isn't a simple way to search the stored values and keys must be known before being used to retrieve or delete associated values. Nevertheless, they are extremely useful as caches of arbitrary values. If you want to try such a database you should visit the try redis tutorial tutorial ~ a gentle yet comprehensive tour of a popular key / value store.

Document oriented databases store arbitrarily structured documents expressed as associative arrays. Every document has a unique ID created by the database and used as a key to reference the document.

However, documents are not opaque values as in key / value databases. Rather, it is possible to reach in to the documents and construct queries that reference the document's named attributes (keys). For example, given a people database I could create a query to retrieve all documents that have a height attribute with a value greater or equal to 180. Since the structure of the documents is arbitrary, such queries rely on conventions (all people documents have a height attribute whose value is expressed in centimetres).

Because document structure is not fixed (except by convention), new fields can be added at any time and document oriented databases specify default handling for missing fields - usually they'll evaluate to null (unknown).

To play with a document oriented database, explore the MongoDB tutorial on the project's homepage.

Relational databases developed from the work of Edgar Codd. They store data in rows (tuples) that share the same columns (attributes) defined in tables (relations) - bracketed words are the formal terminology for the colloquially named concepts. Each row contains a special field called the primary key (the id column in the example) that uniquely identifies the record. Tables, representing different sorts of things, are related to each other with special fields called foreign keys that reference primary keys of rows in different tables (the nationality_id). Many relational databases also enforce constraints such as referential integrity - where foreign keys may not refer to non-existent primary keys.

To work efficiently, data in relational databases should be normalised to avoid duplication. Data need only be stored in one table representing a specific type of thing. Therefore, modifications only happen once and are propagated via the relationships between tables. For example, the nationalities table is the only place where nationalities may be defined. Tables for things with nationalities (such as people) need only reference the single nationalities table via a foreign key rather than duplicate such information. The process of normalisation usually has three steps of increasing efficiency: first, second and third normal forms. Sometimes it is necessary to de-normalise a database for performance reasons - it's faster to retrieve a single row than it is to retrieve several related rows from different tables.

Structured Query Language (SQL) is the programming language used by relational databases to work with data. Based upon relational algebra, it provides CRUD functionality and capabilities for specifying the structure and relationships between data. It's easy to create powerful queries:

SELECT people.first_name, people.surname,
FROM people
JOIN nationalities ON = people.nationality_id
WHERE people.surname = "Jones"

The result of which is:

people.first_name people.surname
Terry Jones Australian

The query asks: give me the first name, surname and nationality (line 1) of people whose surname is "Jones" (line 4). We have to specify where to collect this data (line 2) and how to join the related tables together (line 3). Notice that SQL doesn't specify how to run the query but describes what the expected result should be.

Interactions with a database happen as transactions. Transactions can contain many different commands and should have several desirable properties:

  • Atomicity - transactions are an all-or-nothing affair that cannot be abandoned in an incomplete state.
  • Consistency - transactions fail if they attempt to put the database in to an invalid state such as breaking referential integrity or some other constraint.
  • Isolation - each transaction is independent of and cannot interfere with any of the others.
  • Durability - once complete a transaction's effect will remain despite catastrophic events like power failure, crashes or errors.

A database is ACID compliant if its transactions meet these standards.

The pace of change in data-based technology is fast. Relatively recent trends, such as big data (huge data sets and the technology needed to manage them), deserve their own article. Yet, even the technology described above allows companies and governments to build frighteningly accurate pictures of our lives (how do you think Amazon is able to make such useful suggestions?).

Personally, I find the ubiquitous analysis and measurement of our behaviour inadvertently Orwellian. Furthermore, there is a tendency to trust the results obtained from databases without recognizing the meaning level of data relies on interpretation (of which there are many varieties). Nevertheless, there are great opportunities to do good and I am reminded of the Buddhist proverb,

"To every man is given the key to the gates of heaven; the same key opens the gates of hell."

I hope we choose the right gate.

1499 words (not including code examples).

Programming ~ Concise and Simple

(As with all "concise and simple" articles I assume no prior knowledge of the subject and keep the length to less than 1500 words.)

This is a personal reflection on programming: the creation and curation of instructions that make computers do things. Instructions (also known as "source code" or just "code") are organised into programs (or "software") that fulfil specific tasks. People who write programs are called programmers, developers, software engineers or (colloquially) hackers.


Instructions are expressed in programming languages that make it easier for humans to comprehend what is going on. So-called "high level" languages are easier for humans to work with than "low level" languages that express things in a way that is closer to how the computer works. In all cases programs are transformed from the original programming language into the binary instructions that cause the computer to function in a particular way. This transformation can happen in two ways:

  • The program is "compiled" in to binary instructions by a special program called a compiler. The result is an "executable" run by the end user.
  • The program is "interpreted" by an executable interpreter whose function is to act as a sort of on-the-fly translator of the instructions.

Both mechanisms have strengths and weaknesses. Because compiled programs have already been transformed in to the computer's binary instructions they are often much faster than interpreted programs where such transformations happen as the program is running. Yet interpreted programs offer the flexibility of adapting and changing themselves while they are run by the interpreter which makes it easier to solve certain sorts of programming problem. Sometimes a combination of these techniques is used: A JIT compiler will interpret until certain sets of instructions are obviously more frequently used, at which point only the popular sets of instructions get compiled "just in time" to make them faster.

Sometimes programs work as expected and produce valuable results. More often they do not. The failure to make a computer work usefully is called a bug. There are generally two sorts of bug: the wrong outcome is correctly implemented (the design is wrong) or the expected outcome is wrongly implemented (the instructions are wrong). In any case, because computers are complex machines, making them do something useful is surprisingly hard.

Managing and taming complexity is one of the core tasks of a programmer. A common piece of advice given to junior developers is "kiss" (keep it simple, stupid!), for there are fewer things that could go wrong. As computing pioneer Tony Hoare explains,

"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult."

Personally, I think creating simple and useful software is more like writing succinct yet powerful prose than opaque logical wizardry. This has an important side effect: simple and coherent code is easy to understand. Code that is easy to understand is easy to maintain. Maintaining code is the act of fixing bugs and curating instructions so things work efficiently (also known as re-factoring). In this way a program is easy to improve as bugs are found and new features implemented.

Some languages have been created with the explicit aim of making code easy to read and understand (such as Python). Some developers promote certain development styles that they claim encourages programmers to produce high quality software. A classic example of this is test driven development where developers write a failing test encapsulating a feature of their program before actually implementing the feature itself to make the test eventually pass. Others (such as Donald Knuth) promote a literate style of programming where code is interspersed with natural language explanations that tell the "story" of the code.

Another core skill of a programmer is de-constructing and analysing problems. Often the problem is expressed in a vague and open-ended way and it is the task of the developer to precisely constrain the scope of the problem so they create some useful outcome. A personal example, from my first programming class, illustrates this beautifully: my tutor explained that programs were merely instructions and that we had five minutes to provide instructions telling him how to switch on the classroom's lights. How hard could it be to write such instructions? Take a minute to imagine what you would write (you may assume that the person following them knows about directions, actions and names of things).

I can still remember my list:

  • Stand up.
  • Look around until you locate the light switch.
  • Walk in the direction of the switch.
  • Press the switch with your fingers.
  • Retrace your steps.
  • Sit down.

Of course, I failed miserably. My tutor pointed out that he was already stood up, there were desks between him and the light switch and the damn lights were already on anyway.

Time and resources are also essential elements of software development:

First, the developer needs to have a feel of how best to organise a program. This means analysing their implementation of the algorithm (the instructions at a conceptual level) to ensure that it is both timely and makes efficient use of resources like the computer's memory and CPU. Often this is expressed using big-O notation.

Second, programming usually involves a deadline. Customers need to know how much time it will take to finish software given a certain number of developers and some sort of collaborative development process. I've heard this described as "the cat herding problem". Fred Brooks famously observed that adding developers to a struggling project only slows it down further. Furthermore, clients change their minds mid-project (in addition to providing vague problems like the one mentioned above) and there are always unknown unknowns that add unforeseen time and effort to the project. If this were not enough, how do you coordinate teams of programmers who are all working on the same software? It's like trying to get ten people to co-compose a symphony at the same time.

Happily, developers have created methods and tools to help them work together. Small autonomous "agile" teams are preferred over large managed cohorts of developers. Issue tracking systems make it easy for clients to request and prioritise features, report bugs and give developers a clear audit trail upon which to base development decisions. Code is stored in version control systems that allow developers to track each other's changes and resolve any conflicts in the software (for example, when two people edit the same piece of code). In fact, developers have become so good at this sort of thing that others borrow these tools and techniques: lawyers have been known to collaborate on complex legal documents with a version control system.

Finally, programming is a political activity: making software means defining the laws that dictate how the digital world should be. Often this obvious fact is left unacknowledged. However, incumbent interests affected by the rise of digital culture (for example, the so-called "creative" industries represented by organisations such as the RIAA, MPAA and their ilk) have attempted, with some success, to change the law to limit the use of file sharing software used to distribute pirated content.

Yet, programmers have turned the law to their advantage. Richard Stallman, founder of the Free Software Foundation (FSF), became disillusioned with locked down dis-empowering software so set out four essential freedoms:

  • The freedom to run the program, for any purpose (freedom 0).
  • The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
  • The freedom to redistribute copies so you can help your neighbor (freedom 2).
  • The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

These freedoms are enshrined in the General Public License (GPL) - a license that subverts copyright (that seeks to expand the rights of creators at the expense of the rights of others) in to copyleft (that seeks to expand the rights of everyone). There is one important exception that limits everyone: any modifications to source code covered by the GPL must also be freely available under the GPL. While the GPL has been described as communist and cancerous it is not anti-commerce and (I think intentionally) does not mention any political affiliations. Code covered by the GPL can be sold (see sections 10 and 11). It's how the FSF first raised money for their cause.

There's a sentiment gaining in popularity: program, or be programmed. If you would like to learn to program, there are plenty of resources available online.

1497 words. Image credits: © 2013 the author with this service.

Northackton Magic Marble Machines

This evening I attended another friendly Northackton meeting (my local hackerspace). It was great to see so many people turn out for a "build your own marble machine" workshop. I took along two of my kids to join in the fun: Penelope (11) and Sam (8).

As with the previous useless machine workshop we were joined by friend-of-Northackton and maker-extraordinaire Martin Raynsford who has set himself the task of providing 365 laser cut items in 365 days. He provided us with a nifty marble machine ~ a laser cut wooden mechanism that's easy to assemble with just a craft knife, wood glue and a screw driver. If you email Martin he'll supply the necessary files for you to cut your own mechanism or will sell you the pre-cut kits that we were using. Furthermore, his blog is definitely worth a read and packed full of all sorts of interesting projects (the Escher Lizards particularly caught my eye).

Both Penelope and Sam had a great time to the extent that Penelope was trying to work out how she could attend again despite having a string orchestra rehearsal on Monday evenings.

Here's Sam demonstrating the contraption we built together:

I love how he's so engrossed in tweaking the spiral and chute (especially at the end of the clip). Penelope was also rather distracted by last minute adjustments to her solo effort:

Why don't schools make use of such fun, easy-to-build and interesting models? It certainly beats the plastic key ring I "made" over six weeks when I was at school.

Peer to Peer ~ Concise and Simple

(As with all "concise and simple" articles I assume no prior knowledge of the subject and keep the length to less than 1500 words.)

What is the best way to organise diverse entities co-existing together in large dynamic groups? How are such arrangements created? Who is responsible for making things work?

These are important questions in political philosophy and Internet engineering and show that both subjects, at an abstract level, share much in common. From a political perspective the first question can be answered with different forms of government, the second by investigating concepts like justice, rights and law while the third question concerns the role of power and authority. From a technological perspective the first question concerns "architecture" (how computational resources fit together), the second is answered by process, requirements and design while the third question concerns capability, roles, standards and (again) authority.

This post concerns peer to peer (P2P) architecture ~ where peers of equal status (devices running appropriate software) cooperate in a loose decentralised network for mutual benefit. Peer to peer is the antithesis of hierarchy - where some have elevated status and power over others. The image below illustrates how this changes the way devices connect and communicate. The peer to peer network on the left is simply a collection of devices (in blue) with different numbers of connections between them (in grey). The right hand image is the classic client (blue) / server (red) hierarchy where many clients rely on a single server to fulfil their functionality.

p2p v client/server topology

Sometimes it is important to have hierarchy: it is best that a consultant surgeon is in charge of an operation rather than a well-meaning democracy of volunteers. In an ideal world, hierarchy is best when the obvious skill, knowledge and capabilities of an entity are acknowledged to bring about greater benefit for all. In an ideal world, those with elevated status and authority would have earned it via reliable and consistent public displays of such skill, knowledge and capabilities. In an ideal world, the responsibility and trust associated with status and authority would be a serious yet welcome obligation. Unfortunately, we do not live in an ideal world and hierarchy is often the cause of much that is bad and frustrating.

Some technologies, such as the web, are hierarchical and inadvertently introduce leverage points used to exploit or gain power over users. The web has a client/server architecture where information and the capabilities to change information are firmly within the realm of the server (the website). The client, using a browser, has to request the server deliver information to them and make changes for them. Unfortunately, user's data is often stored and delivered in a way that makes it hard to migrate between services. Personal data is analyzed by websites and sold via targeted advertising or (worse) handed to governments and other third parties without explicit consent. Disruption ensues when websites break: there's no way to get important documents if the server is down. The client/server model is centralisation of power and websites are single points of failure.

Contrast the web with peer to peer architecture where the loose and decentralised organisation of computing resources make it very hard for third parties to control users. This simple idea can be illustrated by two popular yet different peer to peer systems: Bittorrent (for sharing large digital assets such as audio and video files) and Bitcoin (a decentralised digital currency).

For Bittorrent, Bitcoin and many other peer to peer systems the three questions I asked at the start can be answered thus:

  1. The most effective way to organise a diverse and dynamic group of participants is with peer to peer architecture (this can be for both technical and political reasons).
  2. The means of creating such a network is via an open protocol describing the expected behaviour of participants (including checks to ensure correctness).
  3. It is the participant's responsibility that they correctly implement the protocol in order to make the system function correctly.

Bittorrent is an incredibly efficient way to share large files because of its peer to peer architecture. Imagine that you wanted me to copy a paper edition of War and Peace. I'd probably photocopy and give you every page in order. It's how large files are usually downloaded: linearly from server to client. Unfortunately, if I were busy or, worse still, had lost my original you wouldn't get a copy. Bittorrent works differently: it's better if I told you who else had a copy so you could ask them to provide different pieces of the original that you then collated. This is faster because your peers share the work concurrently. Furthermore, if one of the peers failed to deliver their piece you'd notice it was missing and request it from another. Therefore, there is no single point of failure. Finally, if someone else wanted a copy then you could provide pieces even if you'd not yet received the complete work. As a result peers give as well as take value from the network.

In reality, users download a "torrent" file containing details of the resource they're interested in and who to ask for it. They become a node on the bittorrent network and request pieces from the identified peers. As clients download and re-assemble the pieces of the file, others on the network request pieces in turn. Thus, it takes a relatively short time for large files to be delivered. Furthermore, popular files spread around the network and become easier to deliver since there are more peers to provide pieces.

Some websites (most famously the PirateBay) collect torrent files and provide search facilities. Other sites, called "trackers", monitor which peers have what files - although more recent versions of bittorrent allow peers to do this themselves. The video below, made by the creators of bittorrent, provides a good summary:

The superior performance, huge number of users and uncontrollable availability of pirated content has led the incumbent "creative" industry to disrupt illegal sharing through legal, economic and political means (for example, the PirateBay is censored within the UK). This is easier said than done and in no way changes the effectiveness of the Bittorrent network itself. Users quickly find other means of finding torrent files.

Bittorrent uses a peer to peer architecture because it is the best technical solution. The legal, economic and political fallout emerged later. In contrast, Bitcoin had political and economic motivations from the start and peer to peer enables such ends.

Bitcoin is a decentralised digital currency based upon an open protocol. Unlike regular currency, controlled by banks and governments, Bitcoin is designed to have no central issuing authority and instead relies on the machinations of the protocol to regulate balances, transactions and the issuing of currency. Like cash, transactions can be anonymous. Bitcoins can be subdivided into 100-million smaller units called "satoshis" and, at the time of writing, one bitcoin is worth just over £8.82.

Transactions reference Bitcoin addresses created by the potential recipient. A Bitcoin user can have many Bitcoin addresses and may even decide to use one address per transaction. They simply identify the recipient:


Sometimes they are displayed as QR-codes to be scanned by mobile phones:

A bitcoin address

Assuming a positive balance you can transfer bitcoins to me using the Bitcoin address above (you can't have a negative Bitcoin balance). Your Bitcoin software publishes and cryptographically signs the details of the transaction and peers check and verify the details before adding it to a set of recent transactions called a "block". Blocks are distributed around the network so balances and transfers are tracked to avoid double spending. The protocol is designed so attackers would need impossibly large amounts of time and computing power in order to subvert the chain of blocks tracking transactions.

New Bitcoins are "discovered" by peers on the network as a reward for solving the computationally hard problem that is part of the process for creating new transaction blocks. The difficulty of the problem is adjusted by the protocol to ensure that bitcoins are issued at a consistent rate. This process is called "mining" and the supply of bitcoins will never exceed 21 million. When this limit is reached miner's efforts will be rewarded with a small transaction fee. The video below explains Bitcoin from an end user's perspective:

Bitcoin is complex and many details have been hidden in my explanation. However, it is an open and audited technology and hundreds of thousands of people trust it enough to use it every day.

Concerns have been raised about tax-avoidance, anonymous payment for illegal goods and fraud via the use of Bitcoin but this also applies to traditional currencies. Ultimately, the peer to peer nature of Bitcoin is the mechanism for liberating users from the interference of banks and governments. It remains to be seen if Bitcoin will become popular and disrupt the financial system in the same way that Bittorrent has done with content delivery.

Peer to peer systems challenge the status-quo with a decentralising power shift. Furthermore, creating such technology is far easier and more effective than traditional means of political change.

The times, they are a-changin'.


1499 words. Image credits: © 2013 the author.

The World Wide Web ~ Concise and Simple

(As with all "concise and simple" articles I assume no prior knowledge of the subject and keep the length to less than 1500 words.)

The World Wide Web (or just "the web") is a global hypertext system invented by Sir Tim Berners-Lee in 1989. The web is not the Internet. Rather, it is an application that sends messages via the Internet. You're using it now!

A map of the Internet

Hypertext is the core concept of the web: pages containing text and other media reference each other via so-called hyperlinks. In this way, pages connect to form a nebulous web of relationships. Hypertext was coined in the 1960s by Ted Nelson although the basic concept dates back to Vannevar Bush's 1945 article, As We May Think where he describes an intriguing device called the Memex (MEMory indEX). Various pre-web hypertext systems were developed in the 1960s, 70s and 80s, but it was only until the early 1990s that Berners-Lee's version of hypertext became popular.

The web is made with these concepts:

  • Page - a document on the web. Web pages can contain text, images, sound, video and forms (to facilitate limited interactivity).
  • Hyperlink - something in a page that, when activated, takes the user to another page on the web. Often abbreviated to "link".
  • URL - a uniform resource location identifies a page or other resource on the web.
  • Browser - software on the user's computer that downloads web pages, displays them and handles user interactions (such as filling in a form or clicking a link).
  • Server - a device on the Internet that runs software to deliver web pages and other resources to browsers.
  • HTTP - HyperText Transfer Protocol describes how browsers and servers interact.
  • HTML - HyperText Markup Language defines the structure of web pages identifying such things as paragraphs, links and headings.
  • CSS - Cascading Style Sheets describe how a page should look by defining layout, colour scheme, typesetting and other presentation information.
  • Javascript - a ubiquitous programming language for browsers. It defines the behaviour of a page beyond the basics of clicking links or submitting forms.

In the early days of the web HTML was the only way to create pages (CSS and Javascript were later developments). The essence of HTML has not changed over the years:

HTML is written as a series of elements. Elements are defined with named tags between less-than (<) and greater-than (>) symbols. The scope of an element is defined by an opening and closing tag. For example, a heading is defined by the h1 element:


The closing tag is just like the opening one apart from the prepended back-slash (/). The tags are a balanced pair: the scope of the heading element is defined as the text "Hello" and displayed as:


Elements can contain other elements but are not allowed to overlap:

<p>this is <strong>valid</strong></p>


<p>this is <strong>not</p></strong>

HTML has a certain structure: two elements, head and body, are contained within an html element. Information about the document is expressed by elements within the head tags. Elements within the body tags define the structure of the page. For example:

    <title>My Web Page</title>
    <link rel="stylesheet" href="/static/css/styles.css"/>
    <meta name="author" content="Nicholas H.Tollervey"/>
    <p>This is some text enclosed within a paragraph tag.
    I can link to <a href="">other docs</a>
    using an anchor tag.</p>

HTML is currently at version five (HTML5). Actually, HTML5 is a collection of standards. Some add capabilities for embedding media while others add new tags. HTML5 colloquially refers to specifications for new browser behaviour too, such as local data storage.

Browsers were originally simple and merely text based applications, as shown in the screenshot below. Despite these shortcomings the structure of the page is very clear and easy to read (hyperlinks are green).

Old skool web

The web "happens" in two places: the client's browser and the website's server.

Clicking a link or typing in a URL makes your browser send a request over the Internet using HTTP (as described here). The server identified within the URL responds with HTML for the web page, CSS describing how to display it and any Javascript needed for the page to function correctly.

Before rendering the page to the screen the browser transforms the HTML in to the document object model (DOM). The DOM is a data structure used to match HTML elements with CSS rules. Furthermore, snippets of Javascript run at certain points in the page's existence and manipulate the DOM as events occur (such as when the page finishes loading, a button is clicked or a form is submitted).

As you interact with the browser it repeats this process to fetch new pages from the websites you visit. Sometimes your browser posts information back to a website (when you fill in a form). The website responds with a page that reflects the information you submitted.

This happens within the realm of your "client" computer. However, an equally interesting story happens on the server.

In the simplest case web pages are static so the server responds with pre-existing content. However, many websites are dynamic and allow you to change them on-the-fly. For example, as you browse a shopping website you may click buttons that add items to a virtual shopping basket. If you visit the web page that represents your basket you expect it to be up-to-date and list the correct number of items.

For this to happen several things must be in place.

The server should know who you are. You are authenticated in some way, usually with a username and secret password, and authorised to interact with certain pages. For example, only you should be able to view and change your settings. Furthermore, the server must not confuse you with other users, so your interaction with the website is associated with a unique session linked to your identity so the server can track who is making requests.

The server should keep track of state. When you add an item to a shopping cart this should be reflected in the way the website behaves. Such state is stored in a database "backend".

The HTML should reflect the state stored in the database. This is achieved with templates: snippets of HTML combined with data specific to the client. Rendering templates to HTML is "frontend". The following snippet is for a simple shopping basket (anything enclosed within {% raw %}{% %}{% endraw %} is replaced):

{% raw %}<h1>Hi, {% user.username %}!</h1>
<p>Your shopping cart contains:</p>
{% for item in user.cart %}
  <li>{{}} - {{item.description}}</li>
{% endfor %}{% endraw %}

The server should be reliable. Between the front and back ends is a logic layer checking requests, interacting with the database and ensuring the correct HTML is served. This needs to work correctly (imagine if, when clicking a "buy this item" button, a website added two items rather than one).

The server should perform well. A website should quickly handle a great many requests at once. Two common ways to achieve good performance are caching (store and re-use things that take time to compute but don't change that much over time) and load balancing (route incoming calls to a pool of servers so the workload is shared). Popular websites consist of many servers handling specific tasks (such as the database, rendering templates or the logic) hidden behind layers of load balancers and caches.

If this sounds complicated, that's because it is. However, the description given above is a simplification of the amazing engineering that takes place to display a web page on your screen.

However, all is not well.

Browsers implement web-standards in different ways: Internet Explorer is famous for getting things wrong. HTML and CSS are more complex with each revision - gone are the days of a beautifully simple hypertext system. Competing technical standards are promoted for commercial rather than engineering reasons and some are so complex they're practically useless. Alas, Javascript is hopelessly unintuitive and provides developers with plenty of rope with which to hang themselves.

Finally, there are political problems.

You can't publish on the web unless you know how to set up a website or you use a third party service such as a social network. Social networks are hugely popular but turn the web into a handful of walled gardens - the antithesis of the decentralised early web. Furthermore, such websites track and analyse user behaviour and sell it (via advertising). They also provide services for storing user's digital assets (in order to harvest yet more data to sell). As has been pointed out before, this is deeply worrying.

Ultimately, it is a question of who controls the infrastructure. Many users are happy to sacrifice privacy and control for ease of use and convenience. Yet I wonder how many realise that they are tracked and influenced by only a handful of very powerful technology companies.


What of the future? I like to think that William Gibson was right when, in 1996 he said,

[T]he World Wide Web [is] the test pattern for whatever will become the dominant global medium.

(Who knows what will happen next?)

1486 words. Image credit: © 2005 under a Creative Commons license. Image of Lynx browser © 2012 by the author. "Infrastructure" cartoon © Randall Munroe under a Creative Commons license.