The bigger the data, the smarter the city

Kevin Pallett wonders how the dual concepts of big data and the thinking city work in tandem – or even individually, when used in a public transport context.

Big Data. Smart Cities. These are two the latest buzzwords we keep reading about in magazines and online articles and also receiving in our twitter feeds from various priests of this new order. However, my question to these evangelists is quite simple: “where does public transport fit into this Big Data plan in actual design, functionality and implementation terms within a smart city digital environment?

What do we want to achieve? What inputs do we need? What integration tools are required? What are the outputs?

Public Transport as we all know is fundamental in the efficient movement of people in a city or metropolitan area, and the systems behind public transport modes (all modes) are extremely complex in terms of the planning, operation and delivery of good transport services. But, if some of these tub-thumpers really came and had a closer look at what ITS and transportation specialists have done around the world, they might understand what data exists in public transport and ITS systems as well as its associated logic and, by doing this, see how far some cities and authorities have come in adapting, merging and using transport data in some wonderful ways.

The tools that have been used, the integration techniques, the lessons we have learned, these all could truly be adapted in this “Smart City” picture.


Sadly, my first and last experience involving a smart city forum was disappointing as it was clear that market verticals/segments such as health, transport, finance, utilities and retail failed to find the common ground where a grand design in terms of a “smart city” could truly be employed. I mean with meaningful system logic, with “meat on the bone” in terms of creating a real city data hub or exchange, and the benefits that merging, adapting and mixing data from different sectors could achieve, not only for business but for the common citizen as well.

There needs to be a realization that a treasure trove of data exists and its potential for usability and some of the techniques used in ITS system deployment as well as other sectors should really be celebrated as well as more closely examined.

Transport, I fervently believe, is the backbone of any true “Smart City” not only now but for years to come, and I believe as technology advances we will see the imperceptible joining of previously disparate IT-related markets as naturally as transport and retail.

Whatever happens I know it will be interesting, exciting and probably challenging!


The following gives some quick examples of projects I have been involved in with a hope it not only demonstrates how forward-thinking clients are becoming in transport, but also triggers some thought on how we could approach “Smart City” projects by exploring what player industry’s have in data and design terms, and how this data can be merged/adapted and thus used for some useful projects and outputs.

Project Type: Provision of a Public Transport Operations Data Management System For Transport Authority.

Project Aim: An authority wanted to develop and implement a new Data Management System (DMS) to facilitate the monitoring and regulate the performance of initially bus services operated under Public Service Contracts using location and planning system data with an expansion capability to incorporate heavy rail and light rail services. As well as monitoring bus operations, the DMS will play a key role in identifying infrastructure pinch points, using the same Timing Points and bus stop GPS data, from which bottleneck issues may be inferred. Ticketing data will also be incorporated at a later date for all transport modes.

Project Challenges: Bus Operators have different AVL systems, each mode has different planning/AVL data systems. No workflows identified.

Approach: The approach was to effectively utilise two bus operators to shape business rules, data flows , delivery of data to a secure data landing area prior to ETL(Extract, transform and load) processes.

KPI (Key Performance Indicators) and monetisation rules as well as workflows for performance monitoring and payment were incorporated to make the system as automated as possible. The plan for delivery is shown in Figure 1.

158 The bigger the data 2

Fig 1 The Delivery Plan

It was felt that this approach was most logical as following DMS incorporation of the initial two Bus Operators, a reusable model and approach would then exist for new bus operators, and this approach could therefore be followed, albeit with data adaptions in the ETL process due to specific data associated with the other modes, eg. Light Rail and Heavy Rail.

It should be noted that our ultimate objective for national public transport monitoring, was that all multimodal information within the DMS (both planned and actual location data) should be of the same data format and structure. This would allow benefits in terms of measuring the inefficiency of service coordination between modes as well as the ability for data to be reused for more purposeful data visualisation techniques.

As KPIs, workflows and business rules would be common in data structure terms although configurable by each mode in regard to performance rules, therefore a standardised approach could also be adopted as modes are added to the system.

This standardising of processes in terms of the clarification,data identification and business rules are summed up in the following “delivery tree”diagram shown in Fig 2.

158 The bigger the data 3

Fig 2 Delivery Tree ProcessComplete Information and IT Hub For Rail.

Project Type:Complete Information and IT Hub For Rail.

The aim of this project was to increase efficiency for a rail operator using a Service Oriented Architecture that incorporated use of data adaptors from various sub systems.

 In effect this forward thinking customer required a dynamic IT solution so their whole planning process (planning of duty rosters, rolling stock and timetables) was solved dynamically with the ability to optimise each individual planning process in real-time. All operational and business data would be gathered in a Management Information/business intelligence solution granting access to a diverse range of reports and templates as well as personal communication devices for mobile staff such as Train Drivers and Guards also interfacing to the system in real time when for example signing on for duty start.

The Management Information design put forward once again utilised a staging area design, although due to the number of subsystems that were to be utilised, this resulted in the system being sectionalised into 13 logical areas where each area had the ability to exchange and merge data with other areas thus creating dynamic reports through carefully designed templates, as well as with an additional benefit to create ad hoc reports as required by users. These 13 logical areas were:

  •  Turnover and Revenue
  • Cost Of Operations
  • Sickness, Holiday and Absence
  • Health And Safety
  • Passenger Level Monitoring
  • Personalized Employee Information
  • Operational Monitoring
  • Punctuality and Cancellation Log
  • Board Of Directors Portal
  • Maintenance
  • Invoicing
  • Vacancy Monitoring
  • A Bidding Portal for New Franchises

The following diagram in Fig 3 shows the concept of data connectivity within the Management Information System design. Incorporating a rationale of cost per operated kilometre and operated minute also meant that operations could also be monitored effectively in monetary terms.

158 The bigger the data 4

Figure 3: the concept of data connectivity within the Management Information System design

My aim was to put across in this article that in transport and ITS we do have great experience of undertaking system integration and I believe that a lot of the processes that we have employed do certainly fit into a “smart city” approach.

The merging of transport movement data, of emission data with regionalized health data is relevant, the merging of location data from vehicle, passenger handset in relation to the geographical location of a store is relevant and will open up new business possibilities for sure, the use of a digital city wireless network with spare capacity for citizens to report information to the city, receive information from the city surely is relevant.

Maybe its time we had a longer look and started defining some real outputs for a true Smart City.


Kevin Pallett is an independent consultant specializing in public transport technology. He supports authorities, public transport operators, large consultancy companies and manufacturers in the transport and ITS domain. He undertakes work globally from system design through to integration, specification writing, procurement and project delivery in all public transport system technology.


Kevin is also a partner in the new Anglo/Danish company 56º North, offering software and application development specifically for public transport and retail and he can be contacted at