In recent years other industries other than pharmaceuticals have seen the need to have tighter controls on product quality and production. In lay mans terms this sometimes converts to the tracking of product batches using bar code IDs and other methods to ultimately ensure client satisfaction and compliance to standards. Compliancy is now a reality especially in the energy industries as a result of “retailing” and the requirement for accurate information. Regulatory bodies have been formed by governments at State or Federal levels to ensure fair trading and compliancy to standards. From an information perspective this has put operational data commonly available in SCADA system to the forefront of asset and retail manager’s minds. The demand for more information, displayed in specific formats, accessible at any time and catering for anomalies in “production” data has meant SCADA systems have bled in pain as information is sometimes extracted to the detriment of SCADA operational performance.
What is often called MIS (Manufacturing Information Systems or probably more accurately in this context Management Information Systems) are far more complex when business rules are applied to raw data and intelligence laced decisions are woven into the enterprise applications. Historically Business Intelligence was sometimes applied in the ERP (enterprise resource planning) systems however the new breed of smart application is coming from the Automation end of town and its called MES (Manufacturing Execution Systems).
The right tools and architecture selected to match the client’s particular business requirement can allow business process improvements never thought possible to be automated with whatever business rules can be dreamed up. Experience has shown that the planning and design stages of MES almost insists on challenging each business rule as permutations of state and contingencies are built into the execution routines to ensure sustainable production no matter what the raw and final material is.
Changing Owners & Project Inception
In compliance to Vencorp reporting requirements, the APA Group (formally Origin Energy Asset Management) embarked on an ambitious project to automate the gathering of data from field instrumentation including RTUs, Inline Correctors, Flow Meters, Pressure meters and other instrumentation required to manage their wide spread gas reticulation network. With sites all over Brisbane and up to Central Queensland, APA needed to maintain operational performance while upgrading architecture to allow MES functionality and Information System platforms. The new platform and architecture formed the basis to support future growth, allow new business rules and reporting methods and maintainability for years to come.
The initial preliminary conceptual design phase of the APA MES system was developed starting November 2006. Within a few short months several engineers reviewed the existing SCADA Platform, existing manually managed Excel report files, field instrumentation devices, communication infrastructure, computer systems, compliancy requirements for government regulation and improved operational business requirements. During the few months of the conceptual design several versions of the compliance standards were released making it necessary to redevelop and consider how information was to be managed to best meet Vencorp requirements. Sometimes this is referred as shifting goal posts.
The development project, including detailed design and implementation began in late February 2007. The target date for completion was an ambitious 30th June the same year. The client objectives were to:
• Provide a reporting system to achieve compliance with Vencorp reporting requirements.
• Develop flexible reporting that is easily configured by users with minimal training.
• Extend the metering to up to 80 sites cost effectively.
• Increase the measurement of pressure within the reticulation system.
• Re-Design and develop the SCADA system in terms of expansion and performance.
• Facilitate and support system migration.
• Standardise Engineering and Conventions within the SCADA system with the use of site typing and upgrading of RTU’s to utilise time series protocol for improved data resolution and logging.
• Provide a portable Local HMI for onsite maintenance of outstations.
• Implement a SCADA logging system to facilitate tracking of changes and operator logs.
• Provide conceptual design, recommendations, project budgetary estimations and planning for the project.
• Support Origin decision process to proceed with all or some of the recommendations.
The new system went live within one week of the target date but it was far from finished and operational. The new system limped along as it provided some form of operational management with new connectivity and information platforms to comply with the industry and organisation standards. The major challenge with all information systems that bolt onto existing brown field SCADA is dealing with inconsistent configuration conventions from yesterday’s developers. Already recognising this in the conceptual design stages of the planning, PARASYN had introduced tight conventions and standards for the back office (information and execution system platform) systems which immediately annunciated field instrumentation and SCADA inconsistencies. This is not uncommon and is typical in most SCADA systems we see today when they were developed without consideration for scalability and maintainability. Any form of MIS and MES on brown field SCADA and HMI typically causes pain to asset owners and data users as they understand some of their key processes for the first time fixing up and standardising on hardware and controls so that all like devices can be managed in similar fashion. The power of this system was the standardisation of SCADA to the point that back office applications could leverage the predictability of a defined and implemented convention.
Typically the design approach PARASYN aims for is a single iteration with minor reviews and updates after client review and input. This was impossible for the new APA platform. Although the design process of design, check, validate, release and develop was used throughout the project life cycle it was applied to several key processes sometimes many times over. The net affect was redesign occurring throughout design, testing and implementation stages including rollout!
The real power of the standardisation was more realised with the choice of MES Platform. An object oriented tag based application engine was used to manage and manipulate the most challenging part of the data flow; historical event data. The field devices used throughout the system supported time and date stamped data at the field units. This meant that sometimes data could be recovered hours after an event occurred. With the used of radio, ISDN and PSTN services, late data management was core in the design.
The application engine rules were developed to;
• Validate data received discarding out of band information
• Check, alarm, ignore or replace report data based on business rules
• Provide quality tracking of all states including if the data was from field instruments, generated automatically by the MES engine or substituted by a power user.
• Automatically report to Vencorp; the data consumer
• Constantly check for late data and republish report data after late data was received
• Track all operator actions and provide audit tracking without loss of any real, calculated or operator entered data.
Minor Hiccup During Cutover
During the cutover and testing months Origin Energy Asset Management changed ownership. This meant before the system was completely tested the computer systems and communications architecture for all regional sites needed to shift to a new Corporate environment. This was an example of shifting goal posts that required PARASYN’s Flexibility to overcome. Additional engineers scoped and tested the use of NextG technology to provide alternative flexibility and connectivity to remote areas typically managed over PSTN, local private radio and WAN clusters. By this stage the system was well documented and under control allowing a seamless cutover to the new corporate networks. PARASYN Engineers instigated firmware changes to NextG hardware and designed and configured equipment to ensure the live cutover went without any incidents.
“Considering the landscape of the project, the relationship [with PARASYN] was invaluable... PARASYN has the expertise of a larger organisation but the flexibility of a smaller one. A good combination.”
–Hamish Barron, APA Projects Team
What’s in the final box
System Topology
The APA Reticulation Network has multiple RTU networks used for data collection. Systems operate using a combination of Real Time and Event Logged data. Outstation RTUs are periodically polled by the Master RTU for historical data and Outstation RTUs report changed conditions by exception. This is often referred to as Change of State (COS) reporting.
The remote RTU process data is event logged with time of occurrence, date of occurrence and the process value and subsequently reported to the master RTU for data collection and alarm detection. In addition the PARASYN Monita Server developed specifically for this project retrieves data directly from Monatec Monita Units in the field and provides this data to the SCADA & MES system. The Monita units utilise the GPRS and email protocols to transport report data to a central server.
Field sites typically report alarm conditions within a few seconds of an event unless the site communications medium is limited by the PSTN (Public Switched Telephone Network). Overall the system leverages an efficient polling method for outstations making use of time series data. For the conventional radio channels the design allows MTUs (Master Terminal Units) to manage at least 120 RTUs per repeater with an abundance of process data.
The Telecommunications radio network uses conventional licensed radios at 450Mhz. PSTN and an IP based WAN system is used to recover data from RTUs outside of the Radio Infrastructure coverage areas including Central Queensland. In most instances, Cellular IP Routers provide connectivity in area’s where the corporate WAN is not available directly.
Redundancy is built into the system from the RTU to the application engine and data historians. By virtue of using time series protocols (time and date stamped data) the system can recover with zero loss of data if for some reason the computer infrastructure completely fails. If this event were to occur, after restoration the historical data would backfill the system including the time the alarm was originally created.
Daily, the Primary SCADA Server synchronizes its real time clock with the Master RTU real time clock. Subsequently the next time the Master RTU communicates with each field RTU it synchronizes the outstation RTU real time clock to itself. This facilitates system wide time synchronization for all historical trend data, and important criteria for reporting sequence of events and having accurate customer data. Accuracy is in the order of 50 milliseconds.
Daily, the Application Server validates the data against business rules which ensure compliance to government legislation. When raw data is successfully validated its state is updated whilst unsuccessful validation results in a data estimate being calculated enabling full government compliance of daily reporting to occur. Estimated data requires manual intervention from the system operators to validate or update and also helps identify network asset maintenance.
Daily Reports are generated based upon the historical data retrieved and the data generated by the calculation engine. Data retrieved from the field is stored in the historian and retrieved by the calculation engine to generate the reporting requirements. The main thrust of the reporting requirements is compliance to government legislation and to report daily and monthly energy consumption information to various parties.
Report and historical data is complemented by other enterprise database information to associate all business data into one convenient repository. Asset information, process information, compliancy information, SCADA status and reported information is now accessible using conventional relational database connections.
Driver Development
The PARASYN Monita Server extends the Monita data logger range of equipment by providing communications statistics, time zone management, data multiplexing to real time and historical database features. All operations are managed using a single network allowing field data to be recovered and analysed in a timely and controlled fashion. This server was developed and an integral part of the project to provide efficient cost affective methods to manage lower tiered metering sites. Previously these types of sites were a cost burden to monitor using traditional telemetry components.
The system topology incorporates several different technologies including communications media. Some of the technology is illustrated in the network topology diagram that follows. Integral to the design of this system and never evident from two dimension diagrams is the management of data through all levels of the enterprise application. Communications traffic is a result of how much process information is required. It includes what resolution and how responsive the system must be to special events. Predictions of the data payload were an essential consideration for the APA MES system design. The type of data and the management of data quality is an integral element of the back office process historian data repository.
Key Benefits
There are a number of key standout benefits realised by APA with their new enterprise solution.
• Work flow including application engine with full redundancy. The MES workflow engine executes on historical data sets, not just real time. It’s far more than the average MES application. It copes with data being updated after the previous report period and re-reports it. All states are tracked for transactional data tracking and auditing purposes so even re-reported states are known. The architecture of the MES environment is illustrated in the following figure:
• Substitution Interface allows operator intervention when an instrument fails and the data validation process flags a data quality problem.
• New communication media introduced (NextG) and optimisation of WAN data communications.
• Standardisation of instrumentation and data structures to allow the system to scale up for future expansion.
• Automated Business reporting and management of customer information.
• SCADA and instrumentation data being used for client billing and regulatory reporting.
• Incorporating new technology and adapting to unique business processes which changed throughout the project life cycle.
• Leveraging new field devices that previously were not Automation Industry network aware.
Why was this a challenge?
Perhaps this application presents itself as the greatest engineering challenge undertaken by the group of PARASYN engineers to date. With the changing specification and compressed development cycle, precise coordination, regular communication and clear understanding of deliverables and priorities was required to allow the team of 14 technical staff to work closely with APA business delivery and operational staff to reach a favourable outcome. The technology chosen as the application and calculation engine had never been used in this specific way before. The uniqueness of this is in relation to the management and validation of historical late data sets made it technically demanding. Real time applications are challenging enough from a process control perspective however consideration and design for latent conditions adds a whole new dimension.
Client Comments
“Effectiveness, integrity, takes ownership of issues, and don't require ongoing monitoring”
-Hamish Barron, APA Projects Team
“I have pushed [PARASYN’s] recommendations to other gas companies in Australia - however in some cases they have used a local company rather than PARASYN and now have problems.”
-Marshall Holmes, Gas Operations Manager
Today’s Status
Like all large systems, there are fine tuning, process issues and new developments always in play. The APA system is today preparing to leverage introduced standards to amalgamate with another gas distribution system ultimately growing the overall geographic and infrastructure footprint.
During the course of project development the power of the technology and methods of implementation on the chosen platform have made new back office reporting developments straightforward and very cost affective. At times this has been to the surprise of the database administrators. The data, SCADA points (tags) and historian information is all available at the fingertips of the application developers to customise and present data to meet whatever the business requirement might be.
back to Zenith Awards 2008.