It’s a problem that’s often overlooked, and it’s a costly one: spending far more on data requests than is truly necessary.
The good news: it’s mostly a problem you can solve. You need to tighten your internal processes by managing your operations’ setup and users’ entitlements a little.
Very often, users don’t explore all the available feed delivery options to keep costs minimal. Of course, some costs are unavoidable simply because specific kinds of data are needed at specific times.
From what we’ve seen, the higher costs come from a lack of central data management as well as request management.
You want to make sure that the data you already have – if sufficiently up to date – does not get requested again. By placing some parameters around how, why and when requests by your users are made, you can reduce fees. Every single request can incur a cost, and in our experience, a data fee-conscious initial design paired with transparent cost allocation helps a great deal to stop proliferation.
Some data providers penalize you for repeatedly requesting data that could have been requested in one go. After all, this is a burden on their platform’s infrastructure, hence spending a bit of time upfront analyzing which requests could be combined can pay off considerably.
Along the same lines, certain data platforms incentivize the scheduling of deliveries because it enables them to plan their hardware usage more effectively. This is not a matter of days or even hours – it can simply be a few minutes you need to plan repeated requesting in advance. By knowing the cutoff time when the data is truly needed, and by scheduling an even slightly delayed delivery within that period, you can reduce your provider’s delivery fees.
It is paramount to be aware of each data vendor’s billing cycle. Providers often charge for providing data over a certain period, say, a calendar month or quarter. Simply by checking your needs and making sure your universe of interest is cleaned up before the end of that period (rather than waiting until the beginning of the next month) can immediately reduce data fees.
A key thing to consider is the categorization of the data product’s attributes.
Many data providers have some form of data category-based fee model. Let’s say one such category is T&Cs. Once you have requested content from the T&C category, the data product may allow you to request further fields that fall within the same category without extra charge. But if at the same time, you request data from another category, say, corporate actions or pricing or fundamentals, you may incur an additional charge for that newly used data category – even if for the same instrument or issuer, and you are currently interested in only a single field.
It really comes down to being pragmatic about which data you need, and at which point you need it. In other words, look at it from two angles: the fee model of your data provider and your immediate business requirement.
It is not about paying any less than you are obliged to pay for your data requirement.
Instead, think of it as a sort of ‘just-in-time’ approach to data management with multifold benefits: eliminating unnecessary processing overhead on all sides, onboarding the latest and greatest data timely before its usage, and keeping the bottom line on your financial data invoices lower – significantly lower.