Despite a seemingly healthy green glow in your dashboards and exemplary regulatory reports, you can’t help but sense that something is amiss with the data. If this feeling rings true for you, don鈥檛 worry – it may be an indication of bigger issues lurking beneath the surface.

听

You’re not alone. In this blog we’ve taken a look at some of the most influential factors that indicate you’ve got a data quality problem. Why not use these handy pointers as a starting point to dig deeper?

1. You’re getting negative feedback from your internal business partners.

Data is the backbone of any business, so it’s no surprise that a lack of satisfaction from internal partners can often be traced back to data issues. From ensuring quality datasets are delivered at scale, through to solutions aimed towards empowering your colleagues with access to necessary information and context – there are many proactive steps you can take when aiming for better performance in this area. Taking action now will ensure everyone has what they need; fuelling success and transforming negative feedback into positive progress.

2. People keep sending you data in听Microsoft Excel.

Now, we all love Excel. It’s brilliant. It’s made data handling a far more widespread expectation at every level of an organisation. But it does not give any way of source or version controlling your datasets, and is massively prone to its inherent limitations in scale and size. In fact, its ubiquity and almost unilateral adoption means that all your fabulous data lake investments are being totally undermined when things like remediation files, or reports, get downloaded into an Excel sheet. If you’re seeing Excel being used for these kinds of activities, you can bet you’ve a data quality problem (or multiple problems) that are having a real effect on your business.

3.听Your IT team has more tickets than an abandoned car.

If your business teams aren’t getting the data they need, they’re going to keep logging tickets for it. It’s likely these tickets will include:

• Change requests, to get the specific things they need;
• Service requests, for a dataset or sets;
• Issue logs because the data is wrong.

More than an identifier that the data’s not great, this actually shows that the responsibility for accessing and using the data remains in the wrong place. It’s like they’re going to a library with an idea of the plot of the story, and the genre, but they can’t actually search by those terms so they’re stuck in a cycle of guessing, of trial and error.

4. Conclusion

What these indicators have shown is that identifying data quality issues isn’t just for data teams or data observability tools to own. The ablity to recognise that something isn’t right is something that sits just as importantly within business lines, users and teams.听

What to do next is always the key question. Ultimately, data quality can be improved if the right processes and tools are put in place to collect, cleanse, and enrich data. There are several challenges that need to be overcome when dealing with bad data. These challenges include:

• Identifying data quality issues,
• Deploying adequate resources and time to resolve them, and
• Investing in advanced analytical tools.

To do this effectively, enterprise-wide data governance is essential as it provides an actionable framework for businesses to continuously manage their data quality over time. Although implementing changes across an organisation may seem daunting at first, there are a few simple steps which organisations can take today that will help them quickly improve their grip on data quality.

A very important first step is the establishment of a data quality control framework, and helpfully we’ve written about this in the following blog. Happy reading!

听

听

The post How You’ll Know You Still Have a Data Quality Problem appeared first on 糖心传媒.

In this blog post, our Head of Marketing, Matt Flenley, takes a closer look at the latest trends in data and information quality for 2023. He analyses predictions made by Gartner and how they’ve developed in line with expectations to provide insight into the evolution of the market and its various key players. Automation and AI are expected to play a central role in data management, and their impact on the industry will be examined in detail. Additionally, the importance of collaboration and interoperability in a consolidating industry will be highlighted, as well as the potential impact of macroeconomic factors such as labour shortages and recession headwinds on the implementation of these trends. Explore the impact of Data profiling, Data mesh, Data Fabric, and Data Governance on the evolving data management landscape in this analysis.

A recent article by Gartner on predictions and outcomes in technology spend took a fair assessment of market predictions its analysts had made and the extent to which they had developed in line with expectations.  

Rather than simply a headline-grabbing list of the way blockchain or AI will finally rule the world, it鈥檚 a refreshing way to explore how a market has evolved against a backdrop of expected and unexpected developments in the overall data management ecosystem and beyond. 

For instance, while it was known pretty widely that the lessening day-to-day impact of the pandemic would see economies start to reopen, it was harder to predict with certainty that Russia would invade Ukraine to ignite a series of international crises, including cost-of-living, provision of food and energy and a new era of geopolitical turmoil long absent from Europe.  

Additionally, the impacts of the UK鈥檚 decision to leave its customs union and single market with its biggest trading partner were yet to be fully realised as the year commenced. The UK鈥檚 job market has become increasingly challenging for firms attempting to recruit into professional services and technology positions. Reduced spending power in the UK鈥檚 economy, combined with rising inflation and a move into economic recession will no doubt have an impact on organisations鈥� ability and willingness to make capital expenditures.  

In that light, this review and preview will explore a range of topics and themes likely to prove pivotal, as well as the possible impact of macroeconomic nuances on the speed and scale of their implementation.  

1. Automation is the key (but explain your AI!) 

Any time humans have to be involved in extracting, transforming or loading (ETL) of data, it costs a firm time and money, and increases risk. It鈥檚 the same throughout an entire data value chain, wherever there are human hands on it, manipulating it for a downstream use. Human intervention adds value in complex tasks where nuance is required. For tasks which are monotonous or require high throughput, errors can creep in. 

A backdrop of labour shortages, and probable recession headwinds, means that automation is going to be first among equals when it comes to 2023鈥檚 probable market trends. Firms are going to be doing more with less, and finding every opportunity to exploit greater automation offered by their own development teams and the best of what they can find off the shelf. The advantages of this are two-fold: freeing up experts to work on more value added tasks, and reducing the reliance on technical skills which are in high demand.  

Wherever there鈥檚 automation, AI and Machine Learning are not far behind. The deployment of natural language processing has made strides in the past year in automating the extraction of data, tagging and analysis of sentiment, seen in areas such as  speech tagging and entity resolution. The impact of InstructGPT and even more so ChatGPT, late in 2022, demonstrated to a far wider audience both the potency of machine learning and its risks.  

Expect, therefore a massive increase in the world of Explainable AI 鈥� the ability to interpret and understand why an algorithm has reached a decision, and to track models to ensure they don鈥檛 drift away from their intended purpose. The EU AI act is currently working its way through the EU parliament and council, providing the proposed first regulation of AI systems enforcement using a risk-based approach. This will be helpful for firms both building and deploying AI models, providing guidance and application of their use. 

2. Collaborate, interoperate or risk isolation 

In the last few years, there has been significant consolidation across the technologies that collectively make up a fully automated, cloud-enabled, data management platform. Even within those consolidations, such as Precisely鈥檚 multiple acquisitions or Collibra purchasing OwlDQ, the need to expand beyond the specific horizons of these platforms has remained sizeable. Think integration with containerisation solutions like or , or environments such as or , where data is stored, accessed or processed. Consider how many firms leverage Microsoft products by default, so when they release something as significant as for unified data governance, organisations which already offer some or most aspects of a unified data management platform will need to explore how to work alongside as-standard tooling.听

The global trend towards hybrid working has perhaps opened the eyes of many firms outside of large financial enterprises to cloud computing, remote access and the opportunities presented by a distributed workforce. At the same time, it鈥檚 brought their attention to the option to onboard data management tooling from a range of suppliers and based in a wide variety of locations. Such tooling will therefore need to demonstrate interoperability across locales and markets, alongside its immediate home market.  

3. Self-service in a data mesh and data fabric ecosystem 

Like s in a digital age, data mesh and data fabric have arisen as two rival methodologies for accessing, sharing and processing data within an enterprise. However, just as in Shakespeare鈥檚 Verona, there鈥檚 no real reason why they can鈥檛 coexist, and better still, nobody has to stage an elaborate, doomed, poison-related escape plan.听

Forrester鈥檚 Michele Goetz didn鈥檛 hold back in her assessment of the market confusion on this topic in an article well . Both setups are answering the question on everyone鈥檚 lips, which is 鈥渉ow can I make more use of all this data?鈥� The operative word here is 鈥榟ow鈥�, and whether your choice is fabric, mesh or some fun-loving third option stepping into the fight like a data-driven Mercutio, it鈥檚 going to be the decision to make in 2023.  

Handily, the most recent few years has seen a rise in data consultants and their consultancies, augmenting and differentiating from the Big Four-type firms by focusing purely on data strategy and implementation. Data leaders can benefit from working with such firms in scoping Requests for Information (), understanding optimal architectures for their organisation, and happily acknowledging the role of a sage 鈥� or learned Friar 鈥� in guiding their paths.听听

Market trend-wise, those labour shortages referenced earlier have become acutely apparent in the . Alongside the drive towards automation and production machine learning is a growing array of no-code, self-service platforms that business users can leverage without needing programming skill. It is wise therefore to expect further increases in this transition throughout 2023, both in marketing messaging and in user interface and user experience design. 

4. Everyone鈥檚 talking about data governance 

Speaking of data governance, a recent trend has been to acknowledge that firms are embracing that title in order to do anything with data management. Whether it鈥檚 to improve quality, understand lineage, implement master data management or undertake a cloud migration programme, much of this falls to or under the auspices of someone with a .  

The rise of data governance as a function in sectors outside financial services has increase as firms become challenged to do more with their data. At the recent y event in Washington, DC, the vast majority of attendees visiting the 糖心传媒 stand held a data governance role or worked in that area.听听

As a data quality platform provider it was interesting to hear their plans for 2023 and beyond, chiefly around the automation of data quality rules, ease of configuration and needing to interoperate with a wide variety of systems. Many were reluctant to source every aspect of their data management estate from just one vendor, preferring to explore a combination of technologies under an overarching data governance programme, and many were recruiting the specialist services of data governance consultants described previously.  

5. It鈥檚 all about the metadata 

The better your data is classified and quantified with the correct metadata, the more useful it is across an enterprise. This has long been the case, but as in this excellent article on , if anything its reality is only just becoming known. Transitioning from a passive metadata management approach 鈥� storing, classifying, sharing 鈥� to an active one, where the metadata evolves as the data changes, is a big priority for data-driven organisations in 2023. This is especially key in trends such as Data Observability, understanding the impact of data in all its uses and not just in where it came from or where it resides.  

Firms will thus seek technologies and architectures that enable them to actively manage their metadata as it applies to various use cases, such as risk management, business reporting, customer behaviour and so on.  

In the past, one issue affecting the volume of metadata firms could store, and consider being part of an active metadata strategy, was the high cost associated with physical servers and warehouses. However, access to cloud computing has meant that the thorny issue of storing data has, to a certain extent, become far less costly 鈥� lowering the bar for firms to consider pursuing an active metadata management strategy. 

If the cost of access to cloud services was to increase in the coming years, this could be decisive in how aggressively firms explore what their metadata strategy could deliver for them in terms of real-world business results. 

6. And a bonus: Profiling data has never been more important 

Wait, I thought this was a Top 5? Well, on the basis that everyone loves a bit of a January sale, here’s a bonus sixth!

Data profiling is usually the first step in any data management process, discovering exactly what鈥檚 in a dataset. Profiling has become even more pronounced with the advent of production machine learning, and the use of associated models and algorithms. Over the past few years, AI has had a few public run-ins with society, not . For those who missed it, the UK decided to leverage algorithms built on past examination data to provide candidates with a fair predicted grade. However, in reality almost 40% of students received grades lower than anticipated. The data used to provide the algorithm with its inputs were as follows: 

• Historical grade distribution of schools from the previous three years 
• The comparative rank of each student in their school for a specific subject (based on teacher evaluation) 
• The previous exam results for a student for a particular subject 

Thus a student deemed to be halfway in the list in their school would receive a grade equivalent to what the previous halfway pupils achieved in previous years.  

So why was this a profiling issue? Well, for one example, the model didn鈥檛 account for outliers in any given year, making it nigh-on impossible for a student to receive an A in a subject if nobody had achieved one in the previous three years. Profiling of the data in previous years could have identified these gaps and asked questions of the suitability of the algorithm for its intended use. 

Additionally, when the model started to spark outcry in the public domain, profiling the datasets involved would have revealed biases towards smaller school sizes. So while not exclusively a profiling problem, it was something that data profiling, and model drift profiling (discovering how far has the model deviated from its intent) would have helped to prevent. 

This is especially pertinent in the context of evolving data over time. Data doesn鈥檛 stand still, it鈥檚 full of values and terms which adapt and change. Names and addresses change, companies recruit different people, products diversify and adapt. Expect dynamic profiling of both data and data-associated elements, including algorithms, to be increasingly important throughout 2023 and beyond. 

And for more from 糖心传媒, find us on ,  or 

The post Top 5 Trends in Data and Information Quality for 2023 appeared first on 糖心传媒.

What is Self-Service Data Governance?

听

Self Service Data Governance (SSDG) is an enterprise-wide initiative incorporating data disciplines such as听 data lineage, data quality and data analytics. The purpose of SSDG is to empower business users to access and make sense of their data, without relying on IT departments.

The post What is Self-Service Data Governance? appeared first on 糖心传媒.

Data governance feeds into the bigger picture of an organisation鈥檚 data management initiative, incorporating disciplines such as data quality and data stewardship.

Organisations can establish plans and programmes to help manage and monitor their data flows both internally and externally, joining up the dots between different roles within the enterprise and contributing to better overall operations.

Increasingly businesses are investing in technologies to enable more efficient data governance such as data catalogues, data lineage, data quality and master data management tools.听听

The post What is Data Governance? appeared first on 糖心传媒.

Data quality has been a perennial problem for financial institutions for many years, but this needs to change as firms become increasingly reliant on accurate analytics to deliver business opportunity and competitive advantage.

New approaches to data quality can help firms up their game and significantly improve their analytics capability.

Adding the processes, controls and responsibilities of data governance takes them a step further by ensuring the quality and security of data used across the organisation.

If your organisation is falling short of achieving timely and meaningful analytics, or is doing well but could do better, join to find out how to establish the underlying, yet all-important, essentials of data quality and data governance.

Register for this DMI Data Management Insight webinar to find out:

• How to establish data quality
• How to implement effective data governance
• The benefits of combining data quality and governance
• The best technologies and tools for these tasks
• How to ensure accurate and meaningful analytics

You can find more recent content on this topic here.

The post DMI: How to establish data quality and data governance for analytics | 22/04 appeared first on 糖心传媒.

So where should business and data leaders start to confront this problem?

Many firms start off by putting in place Data Governance strategies. The objective is clear: if I have a set of policies and standards, then people will find it easier to conform and drive up the quality. However, in practice this is unlikely to overcome a problem right now; with r to “how long will this Data Governance programme take?” it makes sense to attack it in a different way at the same time. Below, we’ve curated a “top 3” list to help you get started.

What are the steps to creating a data quality driven culture?

1. Pick a quick win that means a lot to your business
2. Get your business teams on-board
3. Focus on measuring quality in a demonstrable way

Firstly, then: picking a quick win. What’s the biggest impact to your business operation right now? As leaders, you’ll doubtless have a big list, but there’s going to be one that sticks out. Maybe it’s a piece of data that’s tripping everyone up when it comes to risk reporting. After all, nobody wants to have to complete risk returns unnecessarily! Perhaps it’s a problem with how customer addresses are formatted across different systems? Whatever the quick win is, get the business teams around the table and agree a rapid timeframe for delivering a step change in how that data is perceived.

That brings us to our next point: getting business teams on-board. In the old days, keeping business users away from data management might have made sense from the point of view of reducing errors in how data was recorded or standardised. But in today’s world, if business teams only know how to complain about data rather than play an active part in managing it, the problem is only going to get worse! If you can get a few internal teams to commit to resolving the thorniest of problems that also happens to be the lowest of all the hanging fruit, then your chances of further success are made all the greater. But they’ll need tooling that doesn’t require them to retrain as programmers, so be mindful of that when it comes to how you’re actually going to manage it.

Lastly, someone is going to have to sign off on your project spend and agree with you that this was a worthwhile use of everyone’s time. Sure, there are standard data quality metrics that all data people know to measure, but what are the business measures that will help prove that this collaboration project has been worth it? Remember, while the business teams have been used to complaining about bad data, they’ve rarely had a chance to be a part of the solution. Maybe it’s a measure of speed to fix data quality problems, together with a proportion of those now being fixed by business teams? If you can show that the programme has shifted the responsibility and the capability to business teams, you’ll be able to demonstrate how the culture of the organisation is becoming more data mature.

What about you? How have you managed to get your culture data-savvy? Or maybe you’re really struggling to get people involved? Send me a message on and we’ll keep the conversation going!

Where can 糖心传媒 help you on your data management journey? We know your time is precious so if you’d like to find out how we can help deliver self-service data quality to your business, simply book 15 minutes of Brendan’s time by choosing a slot

To learn more about how how Data Quality fits into modern Data Ops strategies, catch our webinar as part of 2020’s Data Management Insight event (or read a blog post version here).

The post 3 Steps To Create A Data-Driven Culture: How To Empower Everyone To Be A Data Citizen appeared first on 糖心传媒.

Last September, Kieran Seaward, our Head of Sales, delivered a keynote at the virtual DMS USA on a data powered economy. In his keynote he unpacked:

• The impact of COVID on shifting business priorities, focussing on how existing data management processes and data architectures have changed 鈥� as well as problems encountered along the way听
• Case studies demonstrating best practice approaches to managing data-driven processes and how to create impact and add value in a post COVID era

We are still living with the many challenges presented by COVID-19, including a wide range of changes to the way business can be conducted. At 糖心传媒 we have been really encouraged that engagement with the market is still strong; since March, and the start of many lockdowns, we鈥檝e conducted many hundreds of calls and meetings with clients and prospects to discuss their data management and business plans. This article is based on a lot of our key findings from these calls and reflects the priorities many data-driven firms

Data will power the next phase of the economy 鈥� good or bad?

As global economies look to get back on their feet, it鈥檚 clear that data quality is more important than ever before. Whether it鈥檚 data for citizen health, financial markets, customer or entity data, or any other type, economies and firms will either be powered by a solid platform of high quality, reliable data, or they are going to grow more slowly, built on poor data quality. It鈥檚 not an overstatement to say that what the next phase of economic growth looks like will rely entirely on the decisions and actions we take now.

Kieran鈥檚 keynote was underpinned by the fact there鈥檚 never been a greater need to get your data quality in order. With some firms really grasping this opportunity, change is visible. However, many have encountered the same old problems with the state of their data, and this can inhibit change.

It is necessary to get data quality under control?

Kieran highlighted that pre-pandemic, commented that poor data costs on average between 15-25% of revenue. This figure reaffirms that there is no better time than now to improve the data quality foundation. The long-term future looks bleak if it is built on the quality of data that we have right now!

What is the importance of a foundation of good data quality?

Both before and since the financial crisis of 2008, there have been many conversations reiterating the importance of data quality foundations. Moving forward particularly after COVID-19, a data quality foundation can not only help you get off to a positive start amidst uncertainty, it can also ensure resilience for the future.

Kieran referred to the many conversations that he has had with wealth management, asset management and insurance firms this year. Following hot on the heels of the most innovative investment banks, more and more of these firms are seriously considering what a data governance and data quality framework should deliver.

Redesigning those data architectures that aren鈥檛 delivering

Kieran went on to detail that there have been a large number of firms who have told him that the 鈥榃aterfall鈥� process of logging tickets into software development and IT to improve data quality simply isn鈥檛 moving quickly enough for them, or with the right level of accuracy, and as a result, they鈥檙e evaluating DataOps processes and tooling. Stuart Harvey, CEO at 糖心传媒 spoke in-depth on this at the European Summit, about having every part of the data management journey at the table on data issues, and this is something one client we signed during the UK lockdown is now putting in place as their data management foundation.

On the growth of DataOps-style approaches, Kieran said:

We鈥檝e been encouraged by the number of firms we鈥檝e spoken to who are keen to start moving away from script-based rules. They鈥檙e federating data quality management out from centralised IT and into the business teams so that business users 鈥� who already know the data and what good looks like 鈥� can self-serve for high-quality, ready-to-use data in their line of work. This covers everything from building rules themselves, through interrogating the data itself in dashboards, right down to making or approving fixes in the data within their domain.

What comes first? Data Governance or Data Quality?

Kieran recently wrote a blog on this topic as it he noted that it comes up in every client engagement! To illustrate the importance of data quality, he gave an analogy:

Imagine you are a kid, and your mother has asked you to tidy your bedroom. Your mum returns a few hours later and the room is still a mess – would you get away with saying 鈥�Yes Mum, but look 鈥� I鈥檝e made an inventory of where everything should be.鈥� I imagine the response you鈥檇 get would be something along the lines of 鈥�well done but can you please tidy your room now?!鈥�

Kieran used this story to draw attention to the fact that it is vital to consider data quality first and foremost, as having a large inventory with disorganised data will lead to key data being either inaccessible or difficult to find.

Making a holistic approach to data management

There are a number of building blocks that make up a holistic approach to data management including data quality, master data management, business glossary/data dictionary, metadata management, and so on. As Kieran reiterated in his keynote, using intelligent integration via APIs, it is now possible to build the next generation of data management platform orchestrate by leveraging 鈥榖est of breed鈥� technology components from a multitude of capable vendors. For example,  recently we have explored a more orchestrated approach with vendors like Solidatus, where 糖心传媒 provides data quality and Master Data Management, and Solidatus provides the Governance and Lineage pieces.

Start small, think big

Kieran鈥檚 session reinforced that if you are exploring introducing new capability/ functionality, completing a proof of concept is a well-proven, low-risk means of proving the efficacy of the software and associated professional services. If the scope is well considered and defined on a specific use case that is causing pain, the quick turnaround of results has the potential to create real impact.  This real impact will ultimately help to make the business case a lot stronger.

This is how 糖心传媒 has engaged with the vast majority of our clients and we have successfully delivered remote proof value projects during lockdown that are now being rolled into production.

To use the analogy about tidying your room… Once you start to clean your desk or wardrobe, you can quickly see that cleaning the rest of the room doesn鈥檛 seem as daunting a task, if you break it into chunks.

In conclusion, at 糖心传媒, we believe that it doesn鈥檛 matter where you start, so long as you make the start!

Pre pandemic, MIT Sloan Management Review commented that poor data costs on average between 15-25% of revenue. This figure reaffirms that there is no better time than now to improve the data quality foundation.

Our platform can help you, get in touch today.

In the next blog, we will be unpacking more themes from Kieran鈥檚 keynote 鈥楢 Data-Driven restart鈥�; we will be looking at taking a long-term view; seeking impact that matters, and finally on the budget and implementation approach.

If you want to watch Kieran鈥檚 keynote in full, you can do by checking out this

The post Data will power the next phase of the economy: DMS USA lookback – Part 1 appeared first on 糖心传媒.

It鈥檚 just as easy to say that we wouldn鈥檛 think of ignoring this kind of data. Last week鈥檚 blog looked at the reasons for why measuring data is important for retail banks, but unless there is a strategy taken to react to the results it鈥檚 arguably pretty much meaningless.

Internal product owners, risk and compliance teams all need to use specific and robust data measurements for analytics and innovation; to identify and serve customers; and to comply with the reams of rules and regulations handed down by regulatory bodies. Having identified a way of scoring the data, it would be equally as bizarre to ignore the results.

However, navigating a smooth path in data management is hampered by the landscape being vast, unchartered and increasingly archaic. Many executives of incumbent banks are rightly worried about the stability of their ageing systems and听are finding themselves ill-equipped for a digital marketplace that is evolving with ever-increasing speed.

Key business aims of using data to achieve necessary cost-savings, and grow revenues through intelligent analytics, snarl up against the sheer volume of human and needing to be ploughed into these systems, in an effort to and to reduce the customer impact, regulatory pressure and听painful听bad press .

Meanwhile,听for those who have them, data metrics are听revealing quality problems, and fixing these issues tends to find its way into a once-off project that relies heavily on manual rules and even more manual re-keying into core systems. Very often, such projects have no capacity to continue that analysis and remediation or augmentation into the future, and overtime data that has been fixed at huge cost starts to decay again and the same cycle emerges.

But if your subject matter experts (SMEs) 鈥� 听your regulatory compliance specialists, product owners, marketing analytics professionals 鈥� could have cost-effective access to their data, it could put perfecting data in the hands of those who know what the data should look like and how it can be fixed.

If you install a targeted solution that can access external reference data sources, internal standards such as your data dictionary, and user and department-level information to identify the data owner, you can self-serve to fix the problems as they arise.

This can be done via a combination of SME review and through machine learning technology that evolves to apply remedial activities automatically because the rules created through correcting broken records can contain the information required to fix other records that fail the same rules.

It might sound like futuristic hype 鈥� because AI is so hot right now 鈥� but this is a very practical example of how new technology can address a real and immediate problem, and in doing so complement the bank鈥檚 overarching data governance framework.

It means that the constant push towards optimised customer journeys and propositions, increased regulatory compliance, and IT transformation can rely on regularly-perfected data at a granular, departmental level, rather than lifting and dropping compromised or out-of-date datasets.

Then the current frustration at delays in simply getting to use data can be avoided, and cost-effective, meaningful results for the business can be delivered in days or weeks rather than months or years.

Head over the next part: ‘Build vs Buy 鈥� Off-the-shelf or do-it-yourself? ‘ or click here for part 1 of this blog, covering the need for data quality metrics in retail banking.

Matt Flenley is currently plying his trade as chief analogy provider at . If your data quality is keeping you awake at night, check out Self-Service Data Quality 鈥� our award-winning interactive data quality analysis and reporting tool that is built to be used by business teams who aren’t necessarily programmers.

The post Part 2: Self-service data improvement is the route to better data quality appeared first on 糖心传媒.

In Houston, legions of expert NASA engineers worked alongside the reserve crew to improvise a way of fitting a square type of carbon dioxide filter into a system designed for circular cartridges, using only what the astronauts had on board: space suit hoses, pieces of card, plastic bags and lots of silver duct tape.

Less famously, this fascinating, life-saving听demonstration of ingenuity burned up in the atmosphere shortly after the crew left it behind on their safe return to Earth.

Now, retail banks might never be faced with genuine life and death situations, but they are frequently challenged by problems which draw teams of engineers and operational staff into War Room settings, amid a search for a fix, workaround or hack that will save the day.

When faced with a new regulation or demand of the vast reams of data held by the bank, the temptation can be to follow the same pattern and assemble the consultants, engineers and specialists to try and figure out how to build a solution from the available knowledge, systems and parts.

But what if the answer to the problem could be bought instead 鈥� off-the-shelf and ready to go? What if it was a simple case of plugging in an existing, scoped and developed solution?

The pros and cons of build vs buy

It鈥檚 fair to say that any decision on building or buying a data quality solution needs to satisfactorily answer the following questions:

1. Will this do what I need, or just what I can do?

Your internal programmers and developers are well able to adapt your systems to add new data fields, system requirements or processes, but will these deliver the results the business wants to achieve and to a known level of accuracy? For instance, if your existing data records are not cross-referenced against external data sources to aid de-duplication and augmentation, will it be cost-effective or indeed possible to develop that capability internally, or should a plug-and-play option be considered instead? Then once you have the deployment in place, can your end users adopt the solution themselves or will it have to keep going back to IT for any changes?

2. Will it deploy correctly?

Many IT changes are delayed as existing systems are rarely fully understood by everyone. As they are adapting core (and very frequently, old) systems, building a solution yourself will require a significant level of scoping to comprehend how the processes currently work and establishing that no downstream impacts will occur. Buying a solution to do a specific task can sit alongside core systems and interface to the desired level; that way, differences in how data are used between completely different parts of the business do not have to have an impact on customers or regulatory reporting further downstream.

3. Will it be possible to measure what is happening?

Internal teams will be able to assess the requirements and deliver a solution. After all, that鈥檚 why you continue to hire them! But can they measure the quality of the data, advise on its condition and conformity to your or external standards, and give you guidance as to how to fix anything that doesn鈥檛 fit? If you can鈥檛 view or report on whether the data is improving or deteriorating over time, it makes the objective pretty meaningless. (The first blog looked at the reasons why data quality measurement is something that cannot be ignored in retail banking)

4. What training and support will be required?

Leveraging existing systems should mean little training is required as users are familiar with the current setup. But how will the end-user actually use the solution, what interface will they have? What level of knowledge or interpretative skill will be required? What process manuals need to be written, tested, and maintained? Will buying a solution prove more practical if training manuals are already a part of the offering on the table? Will your existing development teams also provide support or does the external vendor offer this as part of their deployment?

5. What budget is available?

Build-it-yourself can leverage staff already hired in key roles as well as knowledge of operating systems. This can make the build option seem more cost-effective, but if that were truly the case it鈥檚 arguable that the fintech world wouldn鈥檛 exist. Banks would simply do all their own development .

The temptation can be to try and find one solution to solve all problems at an enterprise level, but being specific about one critical issue and running a proof-of-concept can lead to a better understanding of the problem, and the relative benefits and demerits of the built and bought options. Budget ought to also include the cost of what could be incurred if things go wrong: how readily can the change be backed out, and what customer impact could occur?

In summary, the build vs buy decision needs to take into account not just the time it will take to investigate and deploy the change, but also the subsequent time in maintenance and updates for development teams and downstream users.

If your internal solution delivers only what you can do and not what you need to do then some level of a manual workaround, interrogating or running queries may well still be required, and the risk is that your people end up doing what the machine should be doing rather than being freed up to engage in activities that grow the business.

When it came to planning Apollo 14, NASA didn鈥檛 install the electrical-tape workaround; instead, they started afresh with . Life or death may not be on the cards for a bank but profitability, capability and compliance always are; against that backdrop, the choice to build vs buy is rarely simple, but always critical.

Click here for Parts 1 and 2 of this blog, taking an in-depth look at the need for data quality metrics in retail banking.

The post Part 3: Build vs Buy – Off-the-shelf or do-it-yourself? appeared first on 糖心传媒.

In his previous blog on Good Data Culture, our Head of Client Services, Luca Rovesti, discussed taking first steps on the road to data maturity and how to build a data culture. This time he鈥檚 taking a look at some of the biggest challenges of Data Management that arise once those first steps have been made 鈥� and how to overcome them. Want to see more on this topic? Head here.

One benefit of being part of a fast-growing company is the sheer volume and type of projects that we get to be involved in, and the wide range of experiences 鈥� successful and less so 鈥� that we can witness in a short amount of time.

Without a doubt, the most important challenge that rears its head on the data management journey is around complexity. There are so many systems, business processes and requirements of enterprise data that it can be hard to make sense of it all.

Those who get out of the woods fastest are the ones who recognise that there is no magical way of solving things that must be done.

A good example would be the creation of data quality rule dictionaries to play a part in your data governance journey.

Firstly, there is no way that you will know what you need to do as part of your data driven culture efforts unless you go through what you have got.

Although technology can give us a helpful hand in the heavy lifting of raw data, from discovery to categorisation of data sets (data catalogues), the definition of domain-specific rules always requires a degree of human expertise and understanding of the exception management framework.

Subsequently, getting data owners and technical people to contribute to a shared plan that takes the uses of the data and how the technology will fit in is a crucial step in detailing the tasks, problems and activities that will deliver the programme.

Clients we have been talking to are experts in their subject areas. However, they don’t know what 鈥渂est of breed鈥� software and data management systems can deliver. Sometimes, clients find it hard to express what they want to achieve beyond a light-touch digitalisation of a human or semi-automated machine learning process.

The most important thing that we鈥檝e learned along the way is that the best chance of success in delivering a data management programme involves using a technology framework that is both proven in its resilience and flexible in how it can fit into a complex deployment.

From the early days of ‘RegMetrics’ 鈥� a version of our data quality software that was configured for regulatory rules and pushing breaks into a regulatory reporting platform 鈥� we could see how a repeatable, modularised framework provided huge advantages in speed of deployment and positive outcomes in terms of making business decisions.

Using our clients鈥� experiences and demands of technology, we鈥檝e developed a deployment framework that enables rapid delivery of data quality measurement and remediation processes, providing results to senior management that can answer the most significant question in data quality management: what is the return on investing in my big data?

This framework has enabled us to be perfectly equipped to provide expertise on the technology that marries our clients鈥� business knowledge:

• Business user-focused low-code tooling connecting data subject matter experts with powerful tooling to build rules and deploy projects
• Customisable automation that integrates with any type of data source, internal or external
• Remediation clinic so that those who know the data can fix the data efficiently
• 鈥淐hief Data Officer鈥� dashboards provided by integration into off-the-shelf visualisation tools such as Qlik, Tableau, and PowerBI.

Being so close to our clients also means that they have a great deal of exposure and involvement in our development journey.

We have them 鈥榓t the table鈥� when it comes to feature enhancements, partnering with them rather than sell and move on, and involving them in our regular Guest Summit events to foster a sense of the wider 糖心传媒 community.

It鈥檚 a good point to leave this blog, actually, as next time I鈥檒l go into some of those developments and integrations of our 鈥渟elf-service data quality鈥� platform with our data discovery and matching capabilities.

Click here for the latest news from 糖心传媒, or find us on ,  or 

The post Tackling Practical Challenges of a Data Management Programme appeared first on 糖心传媒.

Data Governance or Data Quality is a problem data managers and users will fully understand, and Kieran鈥檚 approach to this is influenced by thousands of hours of conversation with people at all stages of the process, all unified in the desire to get the data right and build a data culture around quality and efficiency.听

Following hot on the heels of banks, we are seeing a lot of buy-side and insurance firms on the road to data maturity and taking a more strategic approach to data quality and data governance, which is great.听 Undertaking a data maturity assessment internally can throw up some much-needed areas of improvement regarding an organization’s data, from establishing a data governance framework, to updating existing data quality initiatives and improving data integrity.听

From what I hear, the听鈥渄ata quality or governance first?鈥澨齝onundrum听is commonly debated by most firms, regardless of听what stage they are at in a data programme rollout.

Business decisions are typically influenced by the need to either prioritise 鈥榯op-down鈥� data governance activities such as creating a data dictionary and business glossary, or 鈥榖ottom-up鈥� data quality activities such as measurement and remediation of company data assets as they exist today from data sources.听 However, achieving a data driven culture relies on both these initiatives existing concurrently.听

In my opinion, these data strategies are not in conflict but complementary and can be tackled in any order, so long as the ultimate goal is a fully unified approach.听听

I could be biased and say those market insights derived from data quality activities can help form the basis of definitions and terms typically stored in governance systems:听

Figure 1 – Data Quality first

However, the same can be said听inversely, data quality systems can benefit from having critical data elements defined and metadata definitions to help shape measurement rules that need to be applied:听

Figure 2 – Data Governance first

The ideal complementary state is that of Data Governance + Data Quality working in perfect unison, i.e. :

• A听Data Governance听system that contains all identified critical data elements as well as definitions to help determine which Data Quality validation rules are applied to ensure they meet the definitions;
• A听Data Quality platform that validates data elements and connects to the governance catalogue to understand who the responsible data scientist or data steward is, in order to push data to them for review and/or remediation of data quality issues.
  The quality platform can then push data quality metrics back into the governance front-end that acts as the central hub/visualization layer displaying data visuals. This either renders data itself or through connectivity to third parties such as Microsoft PowerBI, Tableau, or Qlik.听

Figure 3 – The ideal, balanced state

In the real world, this decision听can鈥檛听be made in isolation of what the business is doing听right听now听with the information they rely on:

• Regulatory reporting teams听have to听build, update and reconfigure reports in听increasingly听tighter听timeframes.
• Data analytics teams are relying on smarter models for prediction and intelligence in order to perform accurate data analysis.
• Risk committees are seeking access to data for the client,听investor,听and board reporting.听听

If the quality of this information听can鈥檛听be guaranteed, or breaks can鈥檛 be easily identified and fixed, all of these teams will keep coming back to IT asking for custom rules, sucking up much-needed programming resources.

Then when听an under-pressure听IT听can鈥檛听deliver in time,听or the requests are conflicting with one another,听the teams will resort to building in SQL or trying to do it via everyone鈥檚 favourite DIY tool, Excel.听

Wherever firms are on their data maturity model or data governance programme, data quality is of paramount importance and can easily run first, last or in parallel. This is something we are used to helping clients and prospects with at various points along that journey, whether it鈥檚 using our self-service data quality & matching platform to drive better data into a regulatory reporting requirement, or facilitating a broad vision to equip an internal 鈥渄ata quality as-a-service鈥� function.

My colleague Luca Rovesti, who heads up our Client Services team, goes more into this in Good Data Culture.听

I鈥檒l be back soon to talk about probably the number one question thrown in at the end of every demo of our software:

What are you doing about AI?

Click here听for the latest news from 糖心传媒, or find us on听,听听or听

The post Data Governance or Data Quality: not always a 鈥榗hicken & egg鈥� problem appeared first on 糖心传媒.

In this article听with听糖心传媒听CTO Alex Brown, Matt Flenley asks about the nature of no-code and lo-code platforms like 糖心传媒鈥� Self-Service Data Quality, and whether they really are a lighter way to enjoy technology?听

The lo-code no-code paradigm can be a bit like . Some people say that it鈥檚 great, it gets the job done, and听these are usually the business subject matter experts who are used to Excel, especially in banks and large government organisations where that鈥檚 the standard data handling tool in use. Technical people, such as software developers who are听 fluent in programming languages and disciplines, look on aghast at听these blocks of functionality that are being chained together in听macro-enabled workbooks听because they听quickly evolve to become monsters!听These听monsters听become听very expensive if not impossible听to听maintain when, inevitably,听changes听are required to support a change in the development environment and data formats.听听

The perfect combination听for these technical people听is something that fits in with the IT rigour around release schedules,听documentation,听and testing 鈥� and just good practices in how you build stuff, making them robust and reusable.

Creating application that are well-tested and can be reused in other projects are quicker and easier to use for new projects, with a product at the end that is more stable. The whole modular approach is where the 糖心传媒 self-service platform听has been听built:听reusable听components that can be听recycled and customised听for rapid, low–risk听development and deployment within a user-friendly lo-code interface.听听

From a business point of view, the driving force behind the lo-code, the no-code approach is about a听tactical way to address specific problems, where the听existing infrastructure isn鈥檛 delivering what the business needs听but the business users aren鈥檛 technical coders.听For example,听a bank or financial firm听might need to capture an additional piece of information听to听meet听a regulatory requirement.听They听might听design and听provide a webform or something similar that听captures and听relays the data into a datastore,听and then into the听firm鈥檚 regulatory reporting framework. This all plays a part in developing efficient business process management.听

This is where no/lo-code comes in as it allows you to do this kind of thing very quickly 鈥� those kinds听of ad-hoc changes you might need to do to听meet a specific deadline or requirement.听

The demand for this will only increase in a post-COVID-19 environment.听For instance, one of our clients mentioned that听at the start of the UK lockdown phase听they needed to rapidly understand what the state of their email addresses was for all their customers to whom they鈥檇 usually write by post.听Their data team of professional developers had rules built in under听two听hours and a听fully operational听interactive dashboard听a day later that their Risk committee could review and track data quality issues and how quickly they were being fixed.

Our听Self-Service Data Quality听platform, for example, is easily used to address the tactical need for data quality or matching without听writing听any听code, or听waiting for central IT to run queries.听You鈥檝e all the drag & drop capability to build rules, data pipelines, matching algorithms and so on without the need for writing any code, allowing you to do a specific job听really quite听quickly. Platforms like this are听extremely good at these tactical use cases where you don鈥檛 want to rip out and rewrite your existing infrastructure, you just need to do this little add-on job to听make it complete to meet a regulatory reporting requirement or specific business requirement.听

Because our platform doesn鈥檛 force you to use a particular persistence layer听or anything like that, it鈥檚 all API-driven and sits on whatever听Master听Data Management platform that you have, it makes it a really flexible tool that is well-suited to these tactical use cases.

This means that the听total cost of ownership for firms听is far lower听because lo-code platforms听offer听a wide range of extensibility to multiple downstream use cases.听Things like听regulatory compliance,听emerging risks, custom听data matching听or even听migration听projects are听the perfect situations听where听one self-service platform can be leveraged for all these things听without听causing huge delays in IT ticketing processes,听or听multiple conflicting requests hitting the central IT team all at once.听

Ultimately, lo or no-code solutions听are likely to听thrive as business teams听discover that they can听get to use the firm鈥檚 data assets听themselves for听faster results,听without tying their IT teams up in knots.

The post No-code & Lo-code: A Lighter Way To Enjoy Tech? appeared first on 糖心传媒.

I started work in banking in the 90鈥檚听as听a programmer,听developing real-time software systems written in C++.听In these good old days,听I鈥檇 be given a specification, I鈥檇 write some code, test and document it. After a few weeks it would be deployed on the trading floor. If my software broke or the requirements changed it would come back to me and I鈥檇 start this听process听all over again. This听鈥榳补迟别谤蹿补濒濒鈥�听approach was slow and, if I鈥檓 honest, apart from the professional pride of not wanting to create buggy code,听I didn鈥檛 feel a lot of ownership for what I鈥檇 created.听

In the last five years a new methodology in software engineering has changed all that 鈥� it鈥檚 called听DevOps,听and听brings a very strategic and听agile听approach to building new software.

More recently DevOps had a baby sister called听DataOps,听and it鈥檚 this subject that I鈥檇 like to talk about today.

Many Chief Data Officers (CDO) and analysts have been impressed by the increased productivity and agility their Chief听Technology听Officer (CTO)听colleagues are seeing through the use of DevOps. Now they鈥檇 like to get in on the act. In the last few months at 糖心传媒 we鈥檝e been talking a lot to CDO clients about their desire to have a more听agile听approach to听data governance听and how DataOps fits into this picture.听听

In these conversations we鈥檝e talked a great deal about the听ownership听of data. A key question is how to associate the measurement and fixing of a piece听of听broken听data with the person most closely responsible for it. In our experience the owner of a piece of data usually makes the best听data steward. These are the people who can positively affect business outcomes through accurate measuring and monitoring of data and is typically a CDO’s role.听

We have seen a strong desire to push data science processes, including data governance and the measurement of actual data quality听(at a听record听level) into the processes and automation that exist in a bank.听

I鈥檇 like to share with you through some simple examples of what we are doing with our investment bank and wealth management clients. I hope that this shows that a听self-service听approach to data quality (with appropriate tooling) can empower highly agile data quality measurement for any company wishing to implement the standard DataOps processes of validation, sorting, aggregation, reporting and reconciliation.听

Roles in DataOps and Data Quality听

We work closely with the people who use the 糖心传媒 platform, the people that are responsible for the governance of data and reporting on its quality. They have titles like Chief Data Officer, Data Quality Manager, Chief Digital Officer and Head of Regulation. These data consumers are responsible for large volumes of often messy data relating to entities, counterparties, financial reference data and transactions. This data does not reside in just one place; it transitions through multiple bank processes. It is sometimes 鈥渁t rest鈥� in a data store and sometimes 鈥渋n motion鈥� as it passes via Extract,听Transform,听Load (ETL)听processes to other systems that live upstream of the point at which it was sourced.听听

For example, a bank might download counterparty information from Companies House to populate its Legal Entity Master. This data is then published out to multiple consuming applications for Know听Your听Customer (KYC), Anti-Money听Laundering (AML)听and Life Cycle Management. In these systems the counterparty records are augmented with information such as a听Legal听Entity听Identifier (LEI), a Bank听Identifier听Code (BIC)听or a ticker symbol.听听

This ability to empower subject matter experts and business users who are not programmers to measure data at rest and in motion has led to the following trends:听

• Ownership:听Data quality management moves from being the sole responsibility of a potentially remote data steward to听all of those who are producing and changing data, encouraging a data driven culture.听
• Federation:听Data quality becomes听everyone鈥檚 job.听Let鈥檚 think about end of day pricing at a bank. The team that owns the securities master will want to test accuracy and completeness of data arriving from a vendor.听The analyst working upstream who takes an end of day price from the securities master to calculate a听volume-weighted average price听(VWAP)听will have different checks relating to the timeliness of information. Finally,听the data scientist upstream of this who uses the VWAP to create predictive analytics. They want to build their own rules to validate data quality.听
• Governance:听A final trend that we are seeing is the tighter integration with standard governance tools. To be effective, self-service data quality听and DataOps require tight integration with the existing systems that hold data dictionaries,听metadata, and lineage information.

Here鈥檚 an illustration of how of how we see 糖心传媒 Self Service Data Quality听(SSDQ) Platform听integrating with DataOps in a high–impact way that you might want to consider in your own data strategy.听

1. Data Governance Team听

First听off,听we offer a set of pre-built dashboards for PowerBI, Tableau and Qlik that allow your data stewards to have rapid access to data quality measurements which relate听just to them. A user in the London office might be enabled to see data for Europe or, perhaps, just data听in听their department. Within just a few clicks a data steward for the Legal Entity Master system could identify all records that are in breach of an听accuracy check听where an LEI is incorrect,听or a听timeliness check听where the LEI has not been revalidated in the听Global LEI Foundation鈥檚听(GLEIF) database inside 12 months.听

2. Data Quality Clinic: Data Remediation听

Data Quality Clinic extends the management dashboard by allowing a bank to听return broken data to its owner for fixing. It effectively quarantines broken records and passes them to the data engineer in a queue, improving data pipelines and overall data governance & data quality. Clinic runs is a web browser and is tightly integrated with information relating to data dictionaries, lineage and third–party sources for validation. Extending our LEI example just now,听I might be the owner of a bunch of entities which have failed an LEI check. Clinic would show me the records in question and highlight the fields in error. It would connect to GLEIF as the source of truth for LEIs and provide me with hints on what to correct. As you鈥檇 expect,听this process can be enhanced by Machine Learning to automate this听entity resolution听process under human supervision. 听

3. FlowDesigner听Studio: Rule creation, documentation, sharing听

FlowDesigner is the rules studio in which the data governance team of super users build, manage, document and source-control rules for the profiling, cleansing and matching of enterprise data. We like to share these rules across our clients so FlowDesigner comes pre-loaded with rules for everything from name听and听address checking to CUSIP听or听ISIN validation.听

4. Data Quality Manager: Connecting to data sources;听scheduling, automating solutions听

This part of the 糖心传媒 platform allows your technology team to connect to data flowing from multiple sources, schedule how rules are applied to data at rest and in–motion. It allows for the sharing and re-use of rules across all parts of your business. We have many clients solving big data problems involving听hundreds of听millions of records using Data Quality Manager听across multiple different environments and data sources, on-premise or in public (or more typically private) cloud.听

Summary: Self-Service Data Quality for DataOps听

Thanks for joining me today as I鈥檝e outlined how self-service data quality is a key part of successful DataOps. CDOs need real-time data quality insights to keep up with business needs while technical architects require a platform that doesn鈥檛 need a huge programming team to support it. If you have听any听questions about听this topic, or how we鈥檝e听approached听it,听then we鈥檇 be glad to talk with you.听Please get in touch below.听

Click听here听for the latest news from 糖心传媒, or find us on听,听听or听

The post Self-Service Data Quality for听DataOps appeared first on 糖心传媒.

Luca Rovesti, Head of Client Services for 糖心传媒, hears this all the time on calls and at data management events, and has taken some time to tie a few common threads together that might just make that hike more bearable, and the peak a little closer, from the work of data stewards to key software features.

Without further ado: Luca Rovesti鈥檚 Healthy Data Management series, episode 1:

Lots of the people we鈥檙e speaking with have spent the last eighteen months working out how to reliably measure their business data, usually against a backdrop of one or more pressures coming from compliance, risk, analytics teams or board-level disquiet about the general state of their data. All of this can impact the way business decisions are made, so it’s critical that data is looked after properly.听

For those who have been progressing well with their ambition to build a data driven culture, they鈥檙e usually people with a plan and the high level buy-in to get it done, with a good level of data stewardship already existing in the organisation. They鈥檝e now moved their data maturity further into the light and can see what鈥檚 right and what鈥檚 wrong when they analyze their data. In order to build a data culture, they鈥檝e managed to get people with access to data to stand up and be counted as data owners or chief data officers. This enables business teams to take ownership of large amounts of data and encourage data driven decision making. Now, they are looking at how they can push the broken data back to be fixed by data analysts and data scientists in a fully traceable, auditable way, in order to improve the quality of the data. The role of a data scientist is paramount here, as they have the power to own and improve the organisations critical data sets.听

The big push we鈥檙e getting from our clients is to help them federate the effort to resolve exceptions. Lots of big data quality improvement programmes, whether undertaken on their own or as part of a broader data governance plan, are throwing up a high number of data errors. The best way to make the exercise worthwhile is to create an environment where end-users can solve problems around broken data 鈥� those who possess strong data literacy skills and know what good looks like.听

The best way to make the exercise worthwhile is to create an environment where end users can solve problems around broken data 鈥� those who possess strong data literacy skills and know what good looks like.

As a result, we鈥檝e been able to accelerate the development of features for our clients around federated exceptions management through integrating our Data Quality Clinic with dashboarding layers for data visuals, for example, PowerBI, Qlik, Tableau etc. We鈥檙e starting to show how firms can use the decisions being made on data remediation as a vast set of training data for machine learning models, which can power predictions on how to fix data and cut the amount of decision-making time manual reviews need to take.

It鈥檚 a million light-years away from compiling lists of data breaks into Excel files and emailing them around department heads, and it鈥檚 understandably in high demand. That said, a small number of firms we speak to are still coming to us because the demand is to do data analytics and make their data work for their money. However, they simply can鈥檛 get senior buy-in for programmes to improve the data quality. I feel for them because a firm that can鈥檛 build a business case for data quality improvement is losing the opportunity to make optimal use of its data assets and is adopting an approach prone to inefficiency and non-compliance risk.听

…a firm that can鈥檛 build a business case for data quality improvement is losing the opportunity to make optimal use of its data assets and is adopting an approach prone to inefficiency and non-compliance risk.

Alongside these requests are enquiries about how data teams can get from their current position – where they can鈥檛 access or use programming language-focused tools in IT and so have built rules themselves in SQL (relying on those with computer science expertise) – to the place where they have a framework for data quality improvement and an automated process to implement it. I鈥檒l go into that in more detail in the next blog.

For more on Data Maturity and Information Governance, click here听for the latest news from 糖心传媒, or find us on听,听听or听

The post The Road to Data Maturity appeared first on 糖心传媒.

As featured , we’ve been strong advocates of a self-service approach to data quality (SSDQ), especially when it comes to regulated data types and wide-ranging demands on a firm’s data assets.

This whitepaper SSDQ, authored by our CTO , goes deeper into the reasons why this approach is so much in demand and explores the functionalities that a fully self-service environment needs to equip business users with rapid access to high-quality data.

In this Self-Service Data Quality whitepaper, we describe trends and technologies bringing data quality functions closer to the data. Self Service Data Quality democratizes data, moving responsibility and control from central IT functions to data teams and SMEs. As a result, greater operational efficiency and higher value data assets can be achieved.听

Download our Whitepaper SSDQ here. For more information on our user friendly听 Self Service Data Quality platform, take a look at our page here.

听

The Changing Landscape of Data Quality-There has been increasing demand for higher听 quality data quality and less data quality issues听 in recent years 鈥� highly regulated sectors dealing with personal data, such as banking, have had a tsunami of financial regulations such as BCBS239, MiFID, FATCA and many more stipulating or implying exacting standards for data and data processes.

Meanwhile, there is a growing trend for more and more firms to become more Data and Analytics (D&A) driven, taking inspiration from Google & Facebook, to monetize their data assets. This increased focus on D&A has been accelerated by easier and lower-cost access to artificial intelligence (AI), machine learning (ML) and business intelligence (BI) visualization technologies.

However, in the now-waning hype of lots of tools and technologies comes the pragmatic realization that unless there is a foundation of good quality reliable data and efficient data preparation, insights derived from AI and analytics may not be actionable. This is where having a modern data management framework is crucial, where organisations can take a look at how they are approaching data governance and data quality.

With AI and ML becoming more of a commodity, and a level playing field, the differentiator is in the data and the quality of the data… To read more see the whitepaper above.

Click听here for more thought leadership pieces from our industry experts at 糖心传媒, or find us on ,听听or听 for the latest news.

The post Beyond data prep – Whitepaper SSDQ appeared first on 糖心传媒.

With  reporting that 59 zetabytes of data has been captured, created, copied and consumed worldwide since 2010, it鈥檚 easy to see the problems that arise when even a fraction of this is incorrect. 

The chaos that can 鈥� and does 鈥� arise can seem totally insurmountable, creating a problem that鈥檚 as unappealing to solve as it is difficult. It can also make it hard for data leaders to really get their people inspired about the art of the possible in finding out what鈥檚 wrong, what good should look like, and how to make a difference. 

This post, then, is designed to help energise, excite and encourage your data people in three easy-to-implement ways that will help deliver your next data management programme and truly change your data culture. 

Firstly: You don鈥檛 need to rip & replace your expensive tech stack 

The major enterprise data management firms are already in situ at many of the world鈥檚 biggest firms, yet people are still complaining about the quality of the data they鈥檙e working with. Any approach to solve this problem can lead C-suite executives to think that buying more software to replace it is just far too costly and risky to achieve. 

Selecting vendors who can work alongside the Informaticas and IBMs of this world is clearly a pragmatic opportunity to independently measure and improve the quality of data right from the business teams. It puts the platform in your hands, so that you and your teams can play an active part in the data flow in the organisation without disrupting the stable enterprise technology stack. 

(And what’s more, we’ve done this many times before).

Secondly, boil a kettle, not the ocean! 

鈥淏oiling the ocean鈥� is a really evocative phrase when it comes to prioritisation and the approach to take 鈥� especially with something as central and fundamental as data quality. Everyone needs high-quality data, even those who are guilty of kidding themselves that they don鈥檛! Heads of Innovation and Change are discovering that they won鈥檛 be able to innovate or change anything . 

Picking something that will make a real difference for someone with access to the big purchasing levers is clearly a great strategy. If a general desire to improve data quality feels like 鈥渂oiling the ocean鈥�, then how about getting customer data right ahead of a new product launch instead? Fixed dimensions of success, a six-week delivery timeframe and a lower-than-you-think budget for a 鈥渢ime & materials鈥� type licence can go a long way to getting that senior stakeholder buy-in for the bigger dreams you have in mind.

Lastly, now witness the power of this fully self-service system 

The last thing your team wants to be doing is manually cleansing, standardising and matching data. There鈥檚 no quicker way of taking the wind out of a data analyst鈥檚 sails than by giving them manual 鈥渄irty data鈥� work. And with up to 80% of their time currently being spent doing exactly that, it鈥檚 clear that this problem isn鈥檛 simply going to go away in the morning. Automated routines and processes that will do this for the analyst are like coming in with a double espresso with extra espresso to start the day; they鈥檒l be flying at the data in a fit of pure delight. Solving one critical problem in a way that鈥檚 designed for business users to self-serve makes perfect sense. Especially if it’s scalable, repeatable and easily accessible, using automations and pre-built logic to save time and effort, it will liberate your data analysts to attack data-driven problems all over the enterprise and truly transform the culture of your organisation. 

How could rethinking data quality make a difference to your organisation? Hit me up on and let’s continue the conversation!

To learn more about how how Self-Service Data Quality is the best approach to developing a next-gen data management strategy, catch our from 2020 with key input from CTO, Alex Brown (or read a blog post version here).

The post Easy Ideas To Get Your People Excited 糖心传媒 Data Quality appeared first on 糖心传媒.

In the same way that you know when you’re not feeling totally on top of your game, your data has tell-tale signs that not everything is working exactly as it ought. So, without further ado, here are three easy ways to perform a data quality check up, and move from feeling as though something’s not right to pinpointing exactly what you need to do.

1) Find and fix the weeds  

Any gardener can tell you that for good crops to grow, you need to keep on top of the weeds. After all, if you let the weeds take root, it can lead to loss of the whole crop. Likewise, even the smallest discrepancies, inaccuracies, or duplicates can throw your data off balance. This means that re-evaluating potential weaknesses and seeking to correct them is key.

But in just the same way as crops and weeds aren’t necessarily easily distinguishable without some green-fingered expertise, you need to involve the people who know what good looks like to address a data quality challenge.

Tom Redman, the “Data Doc”, has a super-handy method for figuring out just how big the data quality problem is. Head on over and have a read , pick a Friday, schedule a Zoom call, open a beer or two and get cracking!

Once you’ve got your business teams to assess the theoretical size of the problem, you’re already in better shape. A data checkup will help you figure out whether improving the data could be achieved by removing data that isn鈥檛 useful, or filling gaps where data is听limited, or making sure that your reports are fine-tuned on the problematic data elements, business areas or teams.

2) Talk to your front line  

Ultimately, the people who deal with your data day-in, day-out, are the ones at the coal face, the front line, who are capturing data at its source and updating data records at a phenomenal rate.

If you were to poll your people on the quality of data, and whether they understand who is responsible for data quality testing, what would the result be? If it doesn’t bear thinking about, then that is pretty much the answer to your question! Find out how you can empower people in your business process to be as engaged in the data quality story as they are in analytics, winning business and managing customer experience (to name but three). A good starting point is to look at your organisations data quality management processes, including data governance. Within this, data stewards can perform tasks such as data profiling and data monitoring using the data quality dimensions of accuracy, timeliness etc. This will be a useful benchmark for members of the data team to measure the quality of their data long term.

Often, accurate data is a result of trained and competent employees. However, the ever-changing nature of data and the increasing rate of regulations has meant that a manual approach isn鈥檛 enough anymore to achieve high quality data. However, it is a good start for the data team to analyse the data at hand. Recognising there is a data quality issue is the first step towards giving your data a health check. Once you have established there is an issue, a data quality solution can be sourced to solve the problem.听听

3) Talk to your customers!   

Of course, quite apart from your internal discussions on whether things are in a good place or not, there’s nothing quite like hearing it from people who rely on you to manage their customer data.

Ask how many poor customer experiences resulted from bad data, or processes that didn’t align with how the information was to be used. Find those situations where a customer was contacted despite actually having died, or a policy was closed without the customer being aware.

Conducting a “five whys” process into poor experiences is a great start. Ask “why was this a result of poor data quality?” five times, coming up with five different answers, and follow those rabbit holes to the source of the problem. Focus on the problems that matter most to customers and build a comprehensive business case that demonstrates just how customer-centric you really are (and what to do next!). In the long term, this will help establish data integrity within your business.

To find out more about what we offer in the data quality and machine learning fields, drop me an email or connect with me on and let’s continue the conversation!

The post 3 Easy Ways To Give Your Data A Check-Up appeared first on 糖心传媒.

We took some time to catch up over an espresso in a caff猫 around the corner from his swanky new Presales Office in Milan (ok, it was over email) and asked him to share what he spoke about at the events: how banks and financial firms can truly make a standard like Data Capability Assessment Model (DCAM) work in practice.

[Ed] Ciao Luca! Quindi cosa hai fatto negli eventi EDMC?

Ciao! But for the benefit of our readers perhaps we’ll do this in English?

Actually that’s easier for us too.

Great! At 糖心传媒 I have been the technical lead in a number of data management projects, putting me in a good position to talk about the 鈥減ractitioner’s view鈥� 鈥� how does the DCAM framework play out in practice? There is a characteristic about 糖心传媒鈥� business model which I believe makes our story interesting: we are a technology provider AND a consulting firm. This combination means we are not only able to advise on the data programmes and overall strategy, but we are also able to implement our advice. We are not only talking about 鈥渉ow it should be鈥�, but also playing an active role in the implementation phase.

I was able to live through very successful projects…and less successful ones! Looking back at these experiences to determine what made certain interactions more successful than others will be the basis of this blog post.

Ok, so what is the DCAM framework anyway?
The EDMC describes the Data Management Capability Assessment Model (DCAM) as “…the industry-standard, best practice framework designed to assist today’s information professional in developing and sustaining a comprehensive data management program.” You can find more out about it on their website, .

Can you tell us a bit more about the data management frameworks you’ve encountered?
I鈥檇 like to borrow some of DCAM鈥檚 terminology to describe to you how the building blocks of this framework typically interact with each other, as it became evident during our conversations.
When 糖心传媒 comes into the game, the high-level group data strategy is already laid out and we are brought into the picture precisely because someone within the organisation is facing challenges in operationalising such a strategy. This has been the case for every large organisation we have worked with: in all my involvements with Tier 1 banks, I am yet to see no theoretical framework for a data strategy. Clearly BCBS 239 principles, published in 2013, had a prominent role in shaping these (you could even spot a bit of copy/pasting from the principles if you read these frameworks carefully!), but the idea is there to accurately manage the data lifecycle and capture the meaning of the information, the relationship between different data points.

So how does this compare with practice?
Translating theory into practice – the implementation phase – is where things get challenging. There are real business cases to be solved and data programs are set up for this purpose. Solving the business case is critical to prove the return on investment of data management activities. This is where the activities can bring measurable efficiency, avoid regulatory fines and operational losses, and dreaded capital lockdowns. Data programmess would have people responsible for making things happen; how interconnected these people are within the organisation and how acquainted they are with the real business pain points can make a material difference in the success of the projects.

What we see is that there are 2 complementary approaches to service the data program:

• • Data Governance is all about creating a data dictionary, understanding its lineage, ownership and entitlement. This is the top-down approach and it defines 鈥渉ow things should be鈥�.

• Data Quality is all about measurement, accountability and remediation of data, according to test conditions and rules. This is the bottom-up approach and it defines 鈥渉ow the data actually is鈥�.

Do these two approaches, Data Quality and Data Governance, intersect neatly then?
We often get asked about the relationship between Data Quality and Data Governance. Can one exist without the other? Which one is more important? Which one should start first?
I mentioned I was going to give the 鈥減ractitioner view鈥� so I鈥檒l answer from past experience: in the most successful projects I have seen, they were both there. Two parallel activities, with a certain degree of overlap, complementing one another. Governance with policies and definitions, data quality with real metrics on how the data is evolving.

I like to think that governance is like a map, and data quality is the position marker that tells us where we are in the map, making it a lot more useful. The technology architecture is of course where all of this logic plugs in, connecting it with real information in the client’s systems or from external, open or proprietary sources.

Can you give us an example of a live deployment?
Sure! As promised, let鈥檚 see how things played out in a successful interaction. We are typically engaging with organisations to enable the Data Quality part of the data management strategy. This means being instrumental in measuring, understanding and improving the information at the basis of data programmes, and I cannot stress enough the importance of the connection with the underlying business case.

I have seen different models working: whether there would be different programs, each supporting a particular business case, or a single program to service them all.
In our most successful interactions at Tier 1 investment banks or major European banks, we could leverage the data quality initiative to support a number of key business activities, such as regulatory reporting, KYC and AML, because all of these activities rely on complete, consistent and accurate data.

Is there anything else you’d add, something that connects the most successful projects perhaps?
Yes, definitely. The single most important thing that can drive a successful project is buy-in from the technical side. The best implementations we have worked on have depended on a close connection between technical (i.e. IT) teams and the owners of the business case from an implementation point of view. It is extremely rare that a financial institution would go through a level of technical restructuring that would allow a complete change of core technology just to support a business case for regulatory reporting, for example.

In the vast majority of the cases the implementation is 鈥減lug-and-play鈥� with the existing technology architecture, which suits our open architecture and deployment approach completely; and let鈥檚 face it: banks’ IT, DBAs and infrastructure teams are always swamped! More than once it happened to me that it was all ready to go in a project except…for that one server…

But you roll with these punches, and you work on good relationships with internal teams, to help smooth out these wrinkles because the downstream benefits of perfected, cleansed data are almost limitless in their use cases. I mean, these guys know their stuff and care hugely about the quality of their systems and data. So yeah, I’d say financial institutions where there is a close connection between senior teams owning the business case; those responsible for the technology architecture; and the downstream users of the data that would be able to convey its business meaning and measureable benefits, are the perfect grounds for getting projects off the ground and into production in rapid time.

Thanks, Luca! Where can the kindly folk of the internet find out more about you, or maybe meet you if they’re so inclined?
Well, they can always email me and I’ll gladly meet up. I’m going to Amsterdam in March as part of the Department for International Trade’s , so I’d be more than happy to catch up in-person there if that suited. I’m pretty easy-going, especially if coffee’s involved…

The post Run (E)DMC – Where Data Quality & Data Governance Collides appeared first on 糖心传媒.