Realtime Technologies Explained

With technology continuing to accelerate the pace of everyday life around the world, realtime technology on the web is fast becoming one of the sector’s biggest growth areas.

For more, we spoke to Phil Leggetter, who works as a developer evangelist at Pusher and runs a consultancy specialising in realtime web software and technology.

Can you tell us a little bit about what realtime technology means to you? Explain it like I’m five.

My focus with realtime technologies is on how they are used within web applications and beyond. The usage of these technologies on the web has lead to what is now known as “the realtime web”. The realtime web, for me, means a few of things:

 

• The instant availability and discoverability of information

In the past we would publish some content on the web – a blog post, website or new web page – and wait hours or days for that content to be ‘spidered’ by search engines so that potentially interested users could discover it. Realtime technologies used on the web now mean that the instant we publish content it is discoverable through search by others. Twitter and Summize (now Twitter search) were the first mainstream example of this. All mainstream search engines now also spider and index content much faster.

 

• PubSub everywhere

In computing the Publisher-Subscriber pattern is a well understood and commonly used concept when creating applications. It provides a way of registering interest (subscribing) in something (often referred to as a topic or subject) and then, as soon as new information becomes available, it is published and delivered to all subscribers.

Take searching Google for “Aliens” as an example; as soon as you you execute that search Google returns the best search results it has for you at that time. But, imagine if after Google returns that page we then have an Independence Day (film) moment, and the sky becomes filled with alien space craft. The way Google search works right now would mean our page wouldn’t update. We’d be left with a list of historical results about UFOs and the Aliens film. However, if we refresh the page, and because Google now indexes content on the web much faster, we’d start to see results about the impending invasion. What I believe should happen is as soon as Google has new information about “Aliens” that information should be published to anybody who has searched for that topic. I believe that by searching you have subscribed to information and future updates.

 

• The instant delivery of information

Once subscriptions have been set up any new information should instantly be delivered to the interested subscribers. Realtime web technologies make this possible. With web browsers this makes the ‘Refresh’ button obsolete.

 

• Increased interaction and engagement

One of the core benefits of the technology is to make it possible to build interactive experiences; interaction between systems, users and systems and between multiple users. From a user’s point of view interactive experiences lead to increased engagement with an application. How much time do many of us spend using Twitter or Facebook? These are two great examples of interactive communication platforms with a heavy reliance on realtime technologies.

I also have to mention Shakey App which was built at Culture Hack Scotland. It is a really great example of a really interactive, cross platform and cross-space engaging application. To understand what I mean by this you can read more about it on Philip Roberts blog (overview and tech details) and in my post about Shakey App.

 

 

What realtime technologies have you worked with?

As I mentioned earlier, my focus has been realtime web technologies and displaying realtime data within web applications (in a web browser). So, back in 2001 my first job was with a company called Caplin Systems who were building a realtime web framework. The underlying realtime web technology they used was classed as a Comet server. During the past 11 years I’ve worked on and off for Caplin (totallying around 5 years) and have worked on Comet client libraries (APIs) in JavaScript, Flash (ActionScript), .NET, Silverlight and Java.

Comet is an event-focused paradigm for simulating bi-direction realtime communication between a client and a server. The key point here is ‘simulating’ because Comet uses HTTP, which is uni-directional. Therefore the bi-directional communication is achieved using two HTTP connections – one for receiving data and the other for sending commands, such as subscription changes, to the server. Due to the simulated nature of Comet and the inconsistent ways it had to be implemented in web browsers Comet is commonly considered a ‘hack’.

During this time I’ve also had some exposure to:

• HTTP Streaming – used in Comet solutions, but also in realtime streaming APIs by companies such as Twitter, Instagram and DataSift
• HTTP Long-Polling – Another HTTP-based communication mechanism used in Comet solutions
• XMPP - A Verbose protocol focusing on chat and user presence.
• BOSH - Bi-directional streams Over Synchronous HTTP and often associated with XMPP.
• PubSubHubbub - PubSub for RSS, but applicable to other types of content too. Checkout Superfeedr
• WebHooks - Something we’ve potentially been doing for a while, but a very powerful way of realtime server to server communication.

Going back to client technologies; until recently Comet was the solution for realtime web client and server communication. But it’s now being superseded by EventSource and WebSockets. These two technologies fall under the HTML5 umbrella and are standardised solutions, answering a clear need defined by the Comet hacks. EventSource is ultimately a uni-directional HTTP Streaming connection between the client and the server. This is fine for just publishing a stream of data. WebSockets are the solution which has answered the prayers of many a web developer since it offers bi-directional full duplex communication between a client and server over a single connection.

Basically, WebSockets are awesome because we finally have a standardised solution for realtime interactive communication between a client and server.

 

What impact do you think realtime will have on the general public in the future?

 

• It’s already happened. People already expect:
  1. To be presented with the most up to date information. This is probably best described as Live Content.
  2. As soon as new information is available it is delivered/pushed to them.
  3. To be offered interactive experiences such as those offered by Facebook and Twitter

  A very clear example of the demand for Live Content are the BBC websites. Although not truly realtime yet (the information still suffers a bit of a delay) most BBC websites now have a ‘live’ aspect to them. ITV News also recently changed their home page to be a “Live news stream” (they use Pusher). This has resulted in site traffic increasing ten-fold.

  For completeness, the most common use cases for the technology at the moment are:

  • Live Content – news streams, sports scores, betting odds, live blogging, dashboards, analytics
  • Activity Streams – e.g. the Facebook wall
  • Chat – probably the the first thing you’ll try and use the technology for if you are a developer
  • 2nd screen experiences – a lot of us sit with our laptops, tablets or phones whilst watching TV. There’s a big push to create apps that provide experiences that complement whatever we’re already watching on our ‘big screen’.
  • Collaborative applications e.g. Google docs
  • Multiplayer games – many of us have played these sorts of games and realtime technologies make building these much more accessible.
  • Building ‘The Internet of Things’ solutions – potentially another over-used buzzword, but now a reality.

  I think this last point is very exciting indeed. I love the way people are using this technology in innovative ways to connect things in the digital and physical space. Here are a few examples:

  • Arduino + Hexbug + WebSockets
  • Interactive Exhibitions/events 1 2
  • Device and Web interaction: Ninja Blocks & reaDIYmate
  • More remote controlled Arduinos

 

Some people ascribe to the traditional architecture of the web and others say technologies like realtime are blasphemy to that gospel. Do you have an opinion? Does it improve upon or challenge the idea of an open web?

The use of realtime doesn’t make general architectural best practices redundant. Applications can still be built with traditional architectural ‘best practices’ and realtime functionality can be progressively added to these architectures. Realtime applications deal with events; new data is now available, data has been updated etc. But in web applications a request to a web server or a user action in the client (e.g. click or mouse move) are also events, so it’s just about applying the best practices we have there more frequently and in the relevant parts of our applications.

Depending on the realtime technologies that are used it may introduce additional architectural considerations around data flow, processing bottlenecks and resources. Scaling realtime infrastructure can be difficult – we’ve seen the Twitter fail whale enough times. But, they were one of the pioneers so the problems and the solutions are now much better understood partially thanks to them.

 

Are there any security implications to realtime technologies that you are aware of?

The security considerations we have when building a standard applications are just as relevant to realtime technologies. There are concerns with realtime around the data content e.g. with social data how do you judge relevance and sentiment? There aren’t any perfect solutions for these problems right now.

 

Phone or browsers? Better together or do you think that people will gradually start using their mobile devices more exclusively? Is this a good or a bad thing?

I don’t think you can say “Phone or Browsers”. I hope that the web browser application on a mobile device will soon become the most used ‘app’. The browser is an amazing application platform and the functionality (APIs) available in browsers is progressing at a fantastic pace – much faster than say, PHP, RoR (ruby on rails), iOS or .NET runtimes.

We’ve already seen a massive move away from desktop PCs to laptops as a main device and we might see tablet devices overtake laptops in homes. Let’s be honest, the majority of people use devices for surfing the Internet and it could be argued that tablets are better (more convenient) for that. It might be that keyboards simply become peripherals for tablets. When it comes to phones I don’t believe they’re ever going to be big enough for true productivity; you can answer email, surf the web and more, but the experience is better on tablet or laptop simply due to form factor.

When building an application we’re still at the point where you need to consider who the user of your application is and the functionality you are trying to offer, then target the right platform. That may be desktop website, mobile site, responsively designed website or native app. For example, there’s no point in building a native iOS application that requires geolocation functionality for a demographic that commonly uses Nokia 5110s.

You also need to consider the costs of building applications. If you want a native Android, native iOS, mobile web and desktop web application there will be increased costs involved for building for each platform. If you need to cover all devices then targeting web is the best choice as nearly every device has a web browser.

 

What type of applications / areas do you think realtime has the greatest potential for?

Initially the simplest use is anywhere that application content changes frequently. Live news, live blogging, activity streams (walls), sports scores and anything with stats.

What is much more interesting, and something that we’ve seen is already a massive draw, is interactive and engaging experienceslike Facebook and Twitter. Humans crave interaction and realtime web technologies make this possible. Facebook and Twitter initially rely on live content in the form of activity streams from friends and things that a user has registered interest in through ‘like’ing, following and retweeting. This updating content then acts as a trigger for interaction; commenting and conversations. Interactivity results in engagement within an application which ultimately leads to spending more time using that application. This potential engagement is the key business driver influencing the use of realtime web technologies.