Keeping Transactional Data Clean
How do we differentiate between functional data and feature-related data?
We are fortunate in the Natural Gas business that pipelines, the Federal Energy Regulatory Commission (FERC) and other entities are always finding a new way for us to conduct business. New technology, new services, new market drivers all contribute to this endless river of change.
To support these changes, the process is that companies will come to the North American Energy Standards Board (NAESB) with requests to add a new code value, a new data element, a new data set or even new standards to support these new capabilities in a standardized manner. This is good and this is the way the process is designed to work.
Imagine you were going on a long river journey in a canoe. You would pack that canoe carefully to contain exactly what you need. Then, friends come along, with good intentions, and bring you an extra blanket, a specialized cooler, a rope specifically designed to aid in traversing portages, etc. Then the canoe that you packed so carefully begins to become unwieldy. It becomes disorganized. Now you have multiple tools to accomplish the same task. How do you solve this and get back to a manageable canoe? In my case, you’d unpack the canoe, pick the best solution for each task, re-pack the canoe and leave the excess behind.
This is where we are with the NAESB datasets today. We began with a set of datasets in 1996 that were as clean and efficient as we could negotiate. Over the subsequent 20 years, companies have made requests to add new capabilities because of their business practices or new market drivers. They were valid requests. But somewhere along the way, our datasets have become unwieldy. We now have 77 data elements in the nomination dataset alone. We cannot continue to add codes, elements, datasets without paying a price.
There are essentials in the datasets that absolutely must be there, such as the receipt location on a nomination. But we have added many elements and codes along the way that warrant re-evaluation. An example would be that we have three different data elements, as three different options, to bid a transportation rate in the nomination. Initially, I thought that the Mutually Agreed and Business Conditional data elements need to be reviewed, but now I realize that all of the data elements in all of the datasets should be reviewed. The essential elements are functional. The other elements are often features that can be identified in the pipeline business process, within service offering descriptions, or contract terms.
We need to unpack the datasets. Lay out all of the elements and codes. Determine exactly what we really need and determine what we can leave behind.