For spatial data managers and providers, the "e" has typically equated to geoportals -- online anchor sites offering a wealth of spatial data and map layers for users to access, view and order. Though the data transparency and collaborative environments created by online spatial distribution systems can improve productivity and efficiency, organizations have often implemented their self-service concept with a notable "e" missing, and that is "e-delivery". Instead, online spatial distribution has typically meant online ordering and offline fulfillment.
"Allowing users to view spatial data online and request needed data sets is a fairly easy proposition," said Sean Simpson, GIS manager for the city of Surrey's engineering department in British Columbia. "However, manually fulfilling those common, repetitive requests can tax resources and inhibit your ability to serve customers efficiently."
Outdated 'Henry Ford' Model
The offline fulfillment model has dominated, in part, because there haven't been Web and GIS tools that are sufficiently robust to remove the human component of extracting, transforming, integrating and distributing data. Particularly problematic has been finding a way to resolve interoperability issues around diverse data formats, which has left many organizations with a "Henry Ford" type of online offer, says Don Murray co-founder of Vancouver-based Safe Software, a spatial "extract-transform-load" (ETL) company."A key challenge to full-service spatial data distribution is finding an efficient way to distribute data in the format or data model that is immediately usable to varied user communities," Murray said. "Different user communities have different needs, and they need to see the data in different ways. That has often led to data offers similar to Henry Ford's approach when he first introduced his automobile line, which was, 'You can have your car in any color as long as it's black.' So users can order data online, but then they are left to their own devices to restructure the data themselves."
Users Expect Immediacy
With the heightened visibility of spatial information shown by high-profile aggregators and distributors like Google Earth, customers from varied user communities look to and often expect that same immediacy from local government and other providers of spatial data. They want to be able to view data online, choose only the data sets and formats they need, order them and then download the files.Indeed, the unrelenting demand for online information "drive-throughs" has challenged organizations to transition their partly self-service online distribution systems to full-service ones -- from viewing to e-delivery. Thanks to substantial improvements in Web technology and GIS tools -- particularly spatial ETL tools -- organizations in Canada, Europe, the United States and New Zealand are triumphing over significant challenges to automate data processing and create dynamic, user-friendly information drive-throughs that quickly and intelligently serve both internal colleagues and external customers.
Surrey's COSMOS Web GIS system allows users to search for data, order only the specific data set they need, pay for it and download it. Europe's transnational LoG-IN system enables a host of users in Belgium, Germany and the UK to instantly and securely share data in whatever format they need. The Coast Survey Office (OSC) of the U.S. National Oceanic and Atmospheric Administration offers GIS-ready nautical charts for immediate download and integration into GIS workflows through its ENC Direct service. And in New Zealand, Landcare Research's geoportal provides environmental data for automatic download in different formats and coordinate systems.
One Unified Driver
Although managers of these systems did not experience the same development challenges, they all were motivated by one primary driver: relieving their limited personnel resources from the time-consuming administrative tasks of processing data requests."In conducting research at the beginning of the LoG-IN project, we discovered that the typical local GIS professional spends 80 percent of their time in processing information requests and only 20 percent in actually updating and improving data layers," said Filip Meuris, head of the e-government department of Flanders, Belgium-based Intercommunale Leiedal, a regional development agency and primary partner for the partly European Union-funded LoG-IN project.
The same administrative burden was wearing on the GIS data providers at Landcare Research, OSC and the city of Surrey. Spurred by the desire to develop a technical solution to a resource challenge, they chose to capitalize on the connectivity of the Web and the data transformation strengths of Safe Software's FME software to offer a better way to create, manage and distribute spatial data. The end result has been freedom -- for users to be truly self sufficient in choosing and receiving customized data sets, and for GIS professionals to be more proactive, augmenting their data wares to only offer the best and most up-to-date data layers.
"Providing an online distribution system based on FME helps people do their own GIS work and incorporate GIS data into their own workflows," Simpson said. "We can now supply requested raster and vector data on the fly. The system restructures it and presents it to users in the way that they can immediately use it. That helps minimize the burden on our department to service the requests. It's a far more responsive and effective way to service customers."
Reaching this e-freedom comes with its share of technical and infrastructure challenges, but managers say that with the right vision, internal commitment, focused partners, political buy-in and distribution model, an automated Web-based spatial distribution is a rewarding reality.
Dealing With Data
One of the first hurdles to address is data "readiness" -- having a clear understanding of what data is available, where and how it is stored and how much of that data will be made available to particular user groups. Understanding end-users' data needs will also help create a more targeted offering."Knowing and understanding your data and what users want to use the data for are challenging requirements to resolve," said Matt Austin, a physical scientist. "Often systems are developed without talking to or involving the end users, so the service is not what they want or is far more complex then what it needs to be."
Closely linked to the data-understanding exercise is the data conversion challenge -- how to serve a multitude of unique data format preferences and software systems.
The 35 local authorities involved in the LoG-IN project, for example, work with more than 80 different formats and seven software systems. And that doesn't include external customers' partiality to certain formats. Trying to find a GIS tool that could handle all that variety automatically and cater to users' format and projection needs can be a daunting proposition.
However, according to Meuris, with FME the data format is a nonissue. Since FME handles all data conversions automatically, local authorities can maintain their spatial data in their own native formats and coordinate systems. And with the openness of the LoG-IN's Generic Information Infrastructure -- based on open standards, XML/GML and Web services -- users can immediately integrate custom-made data to build powerful Web applications using only a browser.
Surrey, CSO and Landcare Research also faced conversion challenges posed by the significant volumes of data they managed and ultimately wanted to offer. For Landcare, meeting this challenge was an exercise in keystrokes, said Niels Hoffmann, a GIS specialist with the agency's Hamilton branch. Staff designed the FME coordinate reprojection and data set clipping workflow by selecting four data "transformers" and connecting these to create a data flow diagram that automatically controls the data transformation.
"FME easily manages translations from very large
data sets such as our Fundamental Soils Layer that includes over 100,000 features," Hoffmann said. "With its extensive list of supported formats, we'll be able to easily extend our support for formats and coordinate systems in the future."
Mapping a Model
The distribution model may look good on paper, but physically building it may be another story. Indeed, getting diverse data object models, software solutions and hardware systems to communicate, integrate and interoperate -- particularly across countries -- can require some technical ingenuity.For the LoG-IN system, all of the local authorities' varied formats and systems needed to work as one. That meant the data-sharing pipeline's structure needed to be like a chameleon, enabling users to freely request data in whatever format, projection or coordinate system they need and then respond accordingly to that specific request. After testing FME technology for one year, Meuris said it proved to be a configurable data transformation chameleon.
"We are very different as organizations," Meuris said. "We have high demands for data manipulation and we need to serve many different user groups -- GIS and [computer-aided design] alike," Meuris said. "With FME, authorities can upload specialized data sets into their database and the software takes them, brings them to the right nodes of the shared server database and puts them in the right coordinate system. It easily extracts, converts and delivers any data requested. It also allows users to automate some data processing tasks."
Though Landcare's existing geospatial portal allowed users to view available data layers and query map features, it didn't offer a provision for automatic download. Hoffmann also chose FME to complete the distribution process.
"FME proves an extremely flexible platform for automated data delivery," he says. "The software's .NET API allowed for seamless integration into our existing architecture, and its internal job request database has also provided a better audit trail for auditing data usage."
For Surrey's COSMOS system, Simpson had the additional issue of integrating e-commerce functionality -- a payment process that was previously done manually by acquiring the customer's credit card number by telephone.
"For us, there was no precedent to follow for an e-commerce mechanism -- no one else was doing this," says Simpson. "We had to develop the right costing model and find shopping cart technology that would hook in to our existing system, FME and our Web interface.
"Of all the integration components, FME was the only given," he added. "We knew it would handle the data extraction, translation and delivery functions without problem."
The "E" in Distribution
An open window to spatial data demands that managers provide sufficient security to quell data policy concerns. ENC Direct, COSMOS, LoG-IN and Landcare Research all adopted security measures to maintain the integrity of the available data. Users of COSMOS, LoG-IN and Landcare Research need to apply for and receive a valid logon, and security features are all built-in to enable managers to readily add new users and to select data availability based on user clearances.Once on the site, however, there's a new challenge -- providing a complete end-to-end system that allows users to readily find the data they want, extract and order only the specific data layers they need and to deliver that data electronically. What that meant for developers was to find an effective way to displace what had been an intensive, behind-the-scenes human task with a completely seamless, automated electronic back-office function.
The common solution for all of these systems was to employ FME technology to provide the electronic clip, zip and ship functionality for the masses.
"Often users are forced to download far more data than they actually need and then they have to filter through to get exactly what they wanted," says OSC's Austin. "And then
sometimes they need to restructure the data themselves into a usable format. Using FME software for ENC Direct, users can zoom in to the area of interest, turn on the data layers they want, choose their preferred data format and coordinate system and download it as a zip file in seconds."
"Although allowing users to only request a specific subset of our entire data is complex by nature, we needed to make the user experience as easy as possible and support that with an intelligent back-end to foster loyalty to the site," Simpson said. "Based on FME, COSMOS allows us to let the customer rule by giving them the power to choose what they want and how they want it and then delivering it to them in a timely fashion, 24-hours a day."
That freedom to choose and e-delivery are also beneficial products of the Landcare and LoG-IN systems.
"The 80 percent manual work done by local authorities previously is now done automatically or in seconds with a few keystrokes," Meuris said. "Removing the time-consuming data-processing tasks from authorities allows them to be far more efficient and productive, and affords them the time to develop applications or services with far more confidence."
These managers have pioneered their way into end-to-end Web-based spatial distribution systems. Breaking through the online ordering and offline fulfillment pattern, they have driven themselves to self-service online spatial distribution systems that truly emphasize the "e". They may choose to add to the data menus at their information drive-throughs, but it's clear that the back-end will continue to fuel the front-end full service offering.
Mary Jo Wagner is a Vancouver-based freelance writer with more than 15 years experience in covering geospatial technology.
