- Văn bản
- Lịch sử
5 Data Governance for Big Data Anal
5 Data Governance for Big Data Analytics:
Considerations for Data Policies and Processes
It should not come as a surprise that in a big data environment, much like any environment, the end users might have concerns about the believability of analytical results. This is particularly true when there is limited visibility into trustworthiness of the data sources. One added challenge is that even if the producers of the data sources are known, the actual derivation of the acquired datasets may still remain opaque. Striving for data trustworthiness has driven the continued development and maturation of processes and tools for data quality assurance, data standardization, and data cleansing. In general, data quality is generally seen as a mature discipline, particularly when the focus is evaluating datasets and applying remedial or corrective actions to data to ensure that the datasets are fit for the purposes for which they were originally intended.
5.1 THE EVOLUTION OF DATA GOVERNANCE
In the past 5 years or so, there have been a number of realizations that have, to some extent, disrupted this perception of “data quality maturity,” namely:
• Correct versus correction: In many environments, tools are used to fix data, not to ensure that the data is valid or correct. What was once considered to be the cutting edge in terms of identifying and then fixing data errors has, to some extent, fallen out of favor in lieu of process-oriented validation, root cause analysis, and remediation.
• Data repurposing: More organizational stakeholders recognize that datasets created for one functional purpose within the enterprise (such as sales, marketing, accounts payable, or procurement to name a few) are used multiple times in different contexts, particularly for reporting and analysis. The implication is that data quality can no longer be measured in terms of “fitness for purpose,” but instead must be evaluated in terms of “fitness for purposes,” taking all downstream uses and quality requirements into account.
• The need for oversight: This realization, which might be considered a follow-on to the first, is that ensuring the usability of data for all purposes requires more comprehensive oversight. Such oversight should include monitored controls incorporated into the system development life cycle and across the application infrastructure.
These realizations lead to the discipline called data governance. Data governance describes the processes for defining corporate data policies, describing processes for operationalizing observance of those policies, along with the organizational structures that include data governance councils and data stewards put in place to monitor, and hopefully ensure compliance with those data policies.
Stated simply, the objective of data governance is to institute the right levels of control to achieve one of three outcomes:
1. Alert: Identify data issues that might have negative business impact.
2. Triage: Prioritize those issues in relation to their corresponding business value drivers.
3. Remediate: Have data stewards take the proper actions when alerted to the existence of those issues.
When focused internally, data governance not only enables a degree of control for data created and shared within an organization, it empowers the data stewards to take corrective action, either through communication with the original data owners or by direct data intervention (i.e., “correcting bad data”) when necessary.
5.2 BIG DATA AND DATA GOVERNANCE
Naturally, concomitant with the desire for measurably high quality information in a big data environment is the inclination to institute “big data governance.” It is naive, however, to assert that when it comes to big data governance one should adopt the traditional approaches to data quality. Furthermore, one cannot assume that just because vendors, system integrators, and consultants stake their claims over big data by stressing the need for “big data quality” that the same methods and tools can be used to monitor, review, and correct data streaming into a big data platform.
Upon examination, the key characteristics of big data analytics are not universally adaptable to the conventional approaches to data quality and data governance. For example, in a traditional approach to data quality, levels of data usability are measured based on the idea of “data quality dimensions,” such as:
• Accuracy, referring to the degree to which the data values are correct.
• Completeness, which specifies the data elements that must have values.
• Consistency of related data values across different data instances.
• Currency, which looks at the “freshness” of the data and whether the values are up to date or not.
• Uniqueness, which specifies that each real-world item is represented once and only once within the dataset.
These types of measures are generally intended to validate data using defined rules, catch any errors when the input does not conform to those rules, and correct recognized errors when the situations allow it. This approach typically targets moderately sized datasets, from known sources, with structured data, with a relatively small set of rules. Operational and analytical applications of limited size can integrate data quality controls, alerts, and corrections, and those corrections will reduce the downstream negative impacts.
5.3 THE DIFFERENCE WITH BIG DATASETS
On the other hand, big datasets neither exhibit these characteristics, nor do they have similar types of business impacts. Big data analytics is generally centered on consuming massive amounts of a combination of structured and unstructured data from both machine-generated and human sources. Much of the analysis is done without considering the business impacts of errors or inconsistencies across the different sources, from where the data originated, or how frequently it is acquired.
Big data applications look at many input streams originating from within and outside the organization, some taken from a variety of social networking streams, syndicated data streams, news feeds, preconfigured search filters, public or open-sourced datasets, sensor networks, or other unstructured data streams. Such diverse datasets resist singular approaches to governance.
When the acquired datasets and data streams originate outside the organization, there is little facility for control over the input. The original sources are often so obfuscated that there is little capacity to even know who created the data in the first place, let alone enable any type of oversight over data creation.
Another issue involves the development and execution model for big data applications. Data analysts are prone to develop their own models in their private sandbox environments. In these cases, the developers often bypass traditional IT and data management channels, opening greater possibilities for inconsistencies with sanctioned IT projects. This is complicated more as datasets are tapped into or downloaded directly without IT’s intervention.
Consistency (or the lack thereof) is probably the most difficult issue. When datasets are created internally and a downstream user recognizes a potential error, that issue can be communicated to the originating system’s owners. The owners then have the opportunity to find the root cause of the problems and then correct the processes that led to the errors.
But with big data systems that absorb massive volumes of data, some of which originates externally, there are limited opportunities to engage process owners to influence modifications or corrections to the source. On the other hand, if you opt to “correct” the recognized data error, you are introducing an inconsistency with the original source, which at worst can lead to incorrect conclusions and flawed decision making.
5.4 BIG DATA OVERSIGHT: FIVE KEY CONCEPTS
The conclusion is that the standard approach to data governance in which data policies defined by an internal governance council direct control of the usability of datasets cannot be universally applied to big data applications. And yet there is definitely a need for some type of oversight that can ensure that the datasets are usable and that the analytic results are trustworthy. One way to address the need for data quality and consistency is to leverage the concept of data policies based on the information quality characteristics that are important to the big data project.
This means considering the intended uses of the results of the analyses and how the inability to exercise any kind of control on the original sources of the information production flow can be mitigated by the users on the consumption side. This approach requires a number of key concepts for data practitioners and business process owners to keep in mind:
• managing consumer data expectations;
• identifying the critical data quality dimensions;
• monitoring consistency of metadata and reference data as a basis for entity extraction;
• repurposing and reinterpretation of data;
• data enrichment and enhancement when possible.
5.4.1 Managing Consumer Data Expectations
There may be a wide variety of users consuming the results of the spectrum of big data analytics applications. Many of these applications use an intersection of available datasets. Analytics applications are supposed to be designed to provide actionable knowledge to create or improve value. The quality of information must be directly related to the ways the business processes are either expected to be improved by better quality data or how ignoring data problems leads to undesired negative impacts, and there may be varied levels of interest in asserting levels of usability and acceptability for acquired datasets by different parties.
This means, for the scope of the different big data analytics projects, you must ascertain these collective user expectations by engaging the different consumers of big data analytics to discuss
Considerations for Data Policies and Processes
It should not come as a surprise that in a big data environment, much like any environment, the end users might have concerns about the believability of analytical results. This is particularly true when there is limited visibility into trustworthiness of the data sources. One added challenge is that even if the producers of the data sources are known, the actual derivation of the acquired datasets may still remain opaque. Striving for data trustworthiness has driven the continued development and maturation of processes and tools for data quality assurance, data standardization, and data cleansing. In general, data quality is generally seen as a mature discipline, particularly when the focus is evaluating datasets and applying remedial or corrective actions to data to ensure that the datasets are fit for the purposes for which they were originally intended.
5.1 THE EVOLUTION OF DATA GOVERNANCE
In the past 5 years or so, there have been a number of realizations that have, to some extent, disrupted this perception of “data quality maturity,” namely:
• Correct versus correction: In many environments, tools are used to fix data, not to ensure that the data is valid or correct. What was once considered to be the cutting edge in terms of identifying and then fixing data errors has, to some extent, fallen out of favor in lieu of process-oriented validation, root cause analysis, and remediation.
• Data repurposing: More organizational stakeholders recognize that datasets created for one functional purpose within the enterprise (such as sales, marketing, accounts payable, or procurement to name a few) are used multiple times in different contexts, particularly for reporting and analysis. The implication is that data quality can no longer be measured in terms of “fitness for purpose,” but instead must be evaluated in terms of “fitness for purposes,” taking all downstream uses and quality requirements into account.
• The need for oversight: This realization, which might be considered a follow-on to the first, is that ensuring the usability of data for all purposes requires more comprehensive oversight. Such oversight should include monitored controls incorporated into the system development life cycle and across the application infrastructure.
These realizations lead to the discipline called data governance. Data governance describes the processes for defining corporate data policies, describing processes for operationalizing observance of those policies, along with the organizational structures that include data governance councils and data stewards put in place to monitor, and hopefully ensure compliance with those data policies.
Stated simply, the objective of data governance is to institute the right levels of control to achieve one of three outcomes:
1. Alert: Identify data issues that might have negative business impact.
2. Triage: Prioritize those issues in relation to their corresponding business value drivers.
3. Remediate: Have data stewards take the proper actions when alerted to the existence of those issues.
When focused internally, data governance not only enables a degree of control for data created and shared within an organization, it empowers the data stewards to take corrective action, either through communication with the original data owners or by direct data intervention (i.e., “correcting bad data”) when necessary.
5.2 BIG DATA AND DATA GOVERNANCE
Naturally, concomitant with the desire for measurably high quality information in a big data environment is the inclination to institute “big data governance.” It is naive, however, to assert that when it comes to big data governance one should adopt the traditional approaches to data quality. Furthermore, one cannot assume that just because vendors, system integrators, and consultants stake their claims over big data by stressing the need for “big data quality” that the same methods and tools can be used to monitor, review, and correct data streaming into a big data platform.
Upon examination, the key characteristics of big data analytics are not universally adaptable to the conventional approaches to data quality and data governance. For example, in a traditional approach to data quality, levels of data usability are measured based on the idea of “data quality dimensions,” such as:
• Accuracy, referring to the degree to which the data values are correct.
• Completeness, which specifies the data elements that must have values.
• Consistency of related data values across different data instances.
• Currency, which looks at the “freshness” of the data and whether the values are up to date or not.
• Uniqueness, which specifies that each real-world item is represented once and only once within the dataset.
These types of measures are generally intended to validate data using defined rules, catch any errors when the input does not conform to those rules, and correct recognized errors when the situations allow it. This approach typically targets moderately sized datasets, from known sources, with structured data, with a relatively small set of rules. Operational and analytical applications of limited size can integrate data quality controls, alerts, and corrections, and those corrections will reduce the downstream negative impacts.
5.3 THE DIFFERENCE WITH BIG DATASETS
On the other hand, big datasets neither exhibit these characteristics, nor do they have similar types of business impacts. Big data analytics is generally centered on consuming massive amounts of a combination of structured and unstructured data from both machine-generated and human sources. Much of the analysis is done without considering the business impacts of errors or inconsistencies across the different sources, from where the data originated, or how frequently it is acquired.
Big data applications look at many input streams originating from within and outside the organization, some taken from a variety of social networking streams, syndicated data streams, news feeds, preconfigured search filters, public or open-sourced datasets, sensor networks, or other unstructured data streams. Such diverse datasets resist singular approaches to governance.
When the acquired datasets and data streams originate outside the organization, there is little facility for control over the input. The original sources are often so obfuscated that there is little capacity to even know who created the data in the first place, let alone enable any type of oversight over data creation.
Another issue involves the development and execution model for big data applications. Data analysts are prone to develop their own models in their private sandbox environments. In these cases, the developers often bypass traditional IT and data management channels, opening greater possibilities for inconsistencies with sanctioned IT projects. This is complicated more as datasets are tapped into or downloaded directly without IT’s intervention.
Consistency (or the lack thereof) is probably the most difficult issue. When datasets are created internally and a downstream user recognizes a potential error, that issue can be communicated to the originating system’s owners. The owners then have the opportunity to find the root cause of the problems and then correct the processes that led to the errors.
But with big data systems that absorb massive volumes of data, some of which originates externally, there are limited opportunities to engage process owners to influence modifications or corrections to the source. On the other hand, if you opt to “correct” the recognized data error, you are introducing an inconsistency with the original source, which at worst can lead to incorrect conclusions and flawed decision making.
5.4 BIG DATA OVERSIGHT: FIVE KEY CONCEPTS
The conclusion is that the standard approach to data governance in which data policies defined by an internal governance council direct control of the usability of datasets cannot be universally applied to big data applications. And yet there is definitely a need for some type of oversight that can ensure that the datasets are usable and that the analytic results are trustworthy. One way to address the need for data quality and consistency is to leverage the concept of data policies based on the information quality characteristics that are important to the big data project.
This means considering the intended uses of the results of the analyses and how the inability to exercise any kind of control on the original sources of the information production flow can be mitigated by the users on the consumption side. This approach requires a number of key concepts for data practitioners and business process owners to keep in mind:
• managing consumer data expectations;
• identifying the critical data quality dimensions;
• monitoring consistency of metadata and reference data as a basis for entity extraction;
• repurposing and reinterpretation of data;
• data enrichment and enhancement when possible.
5.4.1 Managing Consumer Data Expectations
There may be a wide variety of users consuming the results of the spectrum of big data analytics applications. Many of these applications use an intersection of available datasets. Analytics applications are supposed to be designed to provide actionable knowledge to create or improve value. The quality of information must be directly related to the ways the business processes are either expected to be improved by better quality data or how ignoring data problems leads to undesired negative impacts, and there may be varied levels of interest in asserting levels of usability and acceptability for acquired datasets by different parties.
This means, for the scope of the different big data analytics projects, you must ascertain these collective user expectations by engaging the different consumers of big data analytics to discuss
0/5000
5 dữ liệu các quản trị lớn dữ liệu Analytics:Xem xét cho dữ liệu chính sách và quy trìnhNó không nên đến như là một bất ngờ rằng trong một môi trường dữ liệu lớn, giống như bất cứ môi trường, người dùng cuối có thể có mối quan tâm về believability của phân tích kết quả. Điều này đặc biệt đúng khi không có giới hạn khả năng hiển thị các tin cậy của nguồn dữ liệu. Một thêm thách thức là rằng ngay cả khi các nhà sản xuất của các nguồn dữ liệu được biết đến, các derivation thực tế của datasets mua lại có thể vẫn còn mờ. Phấn đấu cho dữ liệu tin cậy đã thúc đẩy phát triển và trưởng thành của quy trình và công cụ để đảm bảo chất lượng dữ liệu, tiêu chuẩn hóa dữ liệu và dữ liệu làm sạch. Nói chung, chất lượng dữ liệu nói chung được xem như là một kỷ luật trưởng thành, đặc biệt là khi tập trung việc đánh giá datasets và áp dụng hành động khắc phục hậu quả hoặc khắc phục cho các dữ liệu để đảm bảo rằng các datasets có phù hợp cho các mục đích mà họ được ban đầu được dự định.5.1 SỰ TIẾN TRIỂN CỦA QUẢN TRỊ DỮ LIỆUTrong 5 năm qua, hay như vậy, đã có một số realizations mà có, để một số phạm vi, phá vỡ nhận thức của "trưởng thành chất lượng của dữ liệu", cụ thể là:• Đúng so với chỉnh: trong nhiều môi trường, công cụ được sử dụng để sửa chữa dữ liệu, không để đảm bảo rằng dữ liệu hợp lệ hoặc chính xác. Những gì một lần được coi là cắt cạnh trong điều khoản của việc xác định và sau đó sửa chữa lỗi dữ liệu đã, để một số phạm vi, rơi ra khỏi lợi thay cho xác nhận quá trình theo định hướng, gốc gây ra các phân tích, và khắc phục.• Dữ liệu repurposing: bên liên quan tổ chức nhiều nhận ra datasets tạo ra cho một mục đích chức năng trong doanh nghiệp (chẳng hạn như bán hàng, tiếp thị, tài khoản nợ phải trả, hoặc mua sắm đến tên một vài) được sử dụng nhiều lần trong ngữ cảnh khác nhau, đặc biệt là cho báo cáo và phân tích. Ngụ ý ở đây là chất lượng dữ liệu không còn có thể được đo lường trong điều khoản của "phù hợp cho mục đích", nhưng thay vào đó phải được đánh giá trong điều khoản của "thể dục cho các mục đích," tham gia tất cả hạ lưu sử dụng và yêu cầu chất lượng vào tài khoản.• Sự cần thiết cho giám sát: nhận thức này, mà có thể được xem xét theo để là người đầu tiên, là rằng đảm bảo khả năng sử dụng dữ liệu cho tất cả các mục đích yêu cầu giám sát toàn diện hơn. Giám sát như vậy nên bao gồm theo dõi điều khiển tích hợp vào hệ thống phát triển đời và trên cơ sở hạ tầng ứng dụng. Những realizations dẫn đến kỷ luật được gọi là dữ liệu quản trị. Quản trị dữ liệu mô tả các quá trình để xác định công ty dữ liệu chính sách, mô tả quá trình cho operationalizing chấp hành các chính sách, cùng với các cấu trúc tổ chức bao gồm hội đồng quản trị dữ liệu và dữ liệu stewards đặt vào nơi để giám sát, và hy vọng đảm bảo tuân thủ những chính sách dữ liệu. Nói đơn giản, mục tiêu của quản trị dữ liệu là viện cấp quyền kiểm soát để đạt được một trong số ba kết quả:1. cảnh báo: Xác định dữ liệu các vấn đề có thể có tác động tiêu cực kinh doanh.2. phân loại: Ưu tiên những vấn đề liên quan đến của trình điều khiển tương ứng giá trị kinh doanh.3. remediate: Có dữ liệu tiếp viên có những hành động thích hợp khi thông báo sự tồn tại của những vấn đề.Khi tập trung trong nội bộ, quản trị dữ liệu không chỉ cho phép một mức độ kiểm soát đối với dữ liệu tạo ra và chia sẻ trong một tổ chức, nó giup nâng cao những stewards dữ liệu để thực hiện hành động sửa sai, hoặc thông qua giao tiếp với các chủ sở hữu dữ liệu gốc hoặc bởi sự can thiệp trực tiếp dữ liệu (tức là, "chỉnh xấu dữ liệu") khi cần thiết.5.2 LỚN DỮ LIỆU VÀ QUẢN LÝ DỮ LIỆUĐương nhiên, concomitant với mong muốn cho thông tin chất lượng chạy cao trong một môi trường dữ liệu lớn là độ nghiêng để viện "quản trị dữ liệu lớn." Nó là ngây thơ, Tuy nhiên, để khẳng định rằng khi nói đến quản trị dữ liệu lớn một nên áp dụng các phương pháp truyền thống để chất lượng dữ liệu. Hơn nữa, một trong những không thể giả định rằng chỉ vì nhà cung cấp, tích hợp hệ thống, và tư vấn cổ phần yêu cầu của mình qua lớn dữ liệu bằng cách nhấn mạnh sự cần thiết cho "chất lượng dữ liệu lớn" cùng một phương pháp và công cụ có thể sử dụng để giám sát, xem xét, và luồng vào một nền tảng lớn dữ liệu dữ liệu chính xác.Sau khi kiểm tra, các đặc điểm chính của lớn dữ liệu analytics không phải phổ thích nghi với phương pháp tiếp cận thông thường để chất lượng dữ liệu và dữ liệu quản trị. Ví dụ, trong một cách tiếp cận truyền thống để chất lượng dữ liệu, mức độ khả năng sử dụng dữ liệu được tính dựa trên ý tưởng của "kích thước chất lượng của dữ liệu", chẳng hạn như:• Tính chính xác, đề cập đến mức độ mà các giá trị dữ liệu là chính xác.• Đầy đủ, xác định các yếu tố dữ liệu phải có giá trị.• Nhất quán của liên quan đến dữ liệu giá trị trên trường hợp dữ liệu khác nhau.• Tiền tệ, mà nhìn vào sự tươi mát"" của các dữ liệu và cho dù các giá trị được cập nhật hay không.• Tính độc đáo, mà xác định rằng mỗi mục thực thế giới hơn một lần và chỉ một lần trong bộ dữ liệu.Những loại các biện pháp nói chung được dự định để xác nhận dữ liệu bằng cách sử dụng quy tắc được xác định, bắt bất kỳ lỗi nào khi đầu vào không phù hợp với những quy tắc này, và chính xác nhận ra lỗi khi các tình huống cho phép nó. Cách tiếp cận này thường nhắm mục tiêu có kích thước vừa phải datasets, từ nguồn được biết đến, với cấu trúc dữ liệu, với một tập các quy tắc tương đối nhỏ. Hoạt động và phân tích ứng dụng giới hạn kích thước có thể tích hợp dữ liệu chất lượng điều khiển, cảnh báo và chỉnh sửa, và sửa chữa những sẽ làm giảm các tác động tiêu cực về phía hạ lưu.5.3 KHÁC BIỆT VỚI LỚN DATASETSMặt khác, lớn datasets cũng thể hiện những đặc điểm, cũng không làm họ có loại tương tự của các tác động kinh doanh. Lớn dữ liệu analytics thường là trung tâm trên tiêu thụ một lượng lớn của một sự kết hợp của cấu trúc và cấu trúc dữ liệu từ cả hai tạo ra máy và con người các nguồn. Hầu hết các phân tích được thực hiện mà không xem xét các tác động kinh doanh của lỗi hoặc mâu thuẫn trên khắp các nguồn khác nhau, từ nơi mà các dữ liệu có nguồn gốc, hoặc mức độ thường xuyên nó được mua lại.Lớn dữ liệu ứng dụng nhìn nhiều dòng đầu vào có nguồn gốc từ bên trong và bên ngoài tổ chức, một số Lấy từ một loạt các xã hội mạng suối, dòng dữ liệu cung cấp thông tin, tin tức nguồn cấp dữ liệu, bộ lọc tìm cấu hình sẵn, datasets công cộng hoặc mã nguồn mở, cảm biến mạng, hoặc dòng dữ liệu không có cấu trúc khác. Như vậy datasets khác nhau chống lại từ phương pháp tiếp cận để quản trị.Khi dòng datasets và dữ liệu mua lại có nguồn gốc bên ngoài tổ chức, có các thiết bị nhỏ cho kiểm soát đầu vào. Những tài liệu gốc thường rất được obfuscated là có khả năng ít thậm chí biết người đã tạo ra các dữ liệu tại địa điểm đầu tiên, hãy để một mình sử bất kỳ loại giám sát trên dữ liệu sáng tạo.Một vấn đề khác liên quan đến mô hình phát triển và thực hiện cho các ứng dụng lớn dữ liệu. Nhà phân tích dữ liệu là dễ bị phát triển riêng của họ mô hình trong môi trường chỗ thử riêng của họ. Trong những trường hợp này, các nhà phát triển thường bỏ qua truyền thống dữ liệu và quản lý kênh, mở lớn hơn khả năng cho các mâu thuẫn với xử phạt nó dự án. Điều này là phức tạp hơn như datasets được khai thác vào hoặc tải về trực tiếp mà không có sự can thiệp.Thống nhất (hoặc thiếu đó) có lẽ là vấn đề khó khăn nhất. Khi datasets được tạo ra trong nội bộ và người dùng hạ lưu công nhận một lỗi tiềm năng, vấn đề đó có thể được truyền đạt đến chủ sở hữu của hệ thống nguồn gốc. Các chủ sở hữu sau đó có cơ hội để tìm nguyên nhân gốc rễ của những vấn đề và sau đó sửa chữa các quá trình đó đã dẫn đến các lỗi.Nhưng với các hệ thống lớn dữ liệu hấp thụ các khối lượng lớn dữ liệu, một số trong đó có nguồn gốc bên ngoài, có những hạn chế cơ hội để tham gia vào quá trình chủ sở hữu để ảnh hưởng đến sửa đổi hoặc sửa đổi để nguồn. Mặt khác, nếu bạn chọn để "sửa chữa" lỗi dữ liệu được công nhận, bạn đang giới thiệu một mâu thuẫn với nguồn gốc, lúc tồi tệ nhất có thể dẫn đến kết luận không chính xác và thiếu sót ra quyết định.5.4 DỮ LIỆU LỚN GIÁM SÁT: KHÁI NIỆM QUAN TRỌNG NĂMKết luận là rằng các phương pháp tiêu chuẩn để quản trị dữ liệu trong đó dữ liệu chính sách được xác định bởi một hội đồng quản trị nội bộ kiểm soát trực tiếp của khả năng sử dụng của datasets không thể phổ biến áp dụng cho dữ liệu lớn ứng dụng. Và chưa có chắc chắn là một nhu cầu cho một số loại giám sát có thể đảm bảo rằng các datasets là có thể sử dụng và kết quả phân tích là đáng tin cậy. Một cách để giải quyết sự cần thiết cho dữ liệu chất lượng và tính nhất quán là để tận dụng các khái niệm về chính sách dữ liệu dựa vào các đặc tính chất lượng thông tin quan trọng đối với dự án lớn dữ liệu.Điều này có nghĩa là xem xét dự định sử dụng của các kết quả của những phân tích và làm thế nào không có khả năng thực hiện bất kỳ hình thức nào của các điều khiển trên các nguồn của lưu lượng sản xuất thông tin có thể được giảm nhẹ bởi người dùng bên tiêu thụ. Cách tiếp cận này yêu cầu một số các khái niệm then chốt cho dữ liệu học viên và chủ doanh nghiệp quá trình để ghi nhớ:• quản lý người tiêu dùng mong đợi dữ liệu;• xác định kích thước chất lượng dữ liệu quan trọng;• Giám sát sự nhất quán của dữ liệu siêu dữ liệu và tham khảo làm cơ sở để thực thể tách;• repurposing và reinterpretation của dữ liệu;• dữ liệu làm giàu và nâng cao khi có thể.5.4.1 quản lý sự mong đợi dữ liệu tiêu dùngCó thể có nhiều người sử dụng tiêu thụ các kết quả của quang phổ của ứng dụng phân tích dữ liệu lớn. Nhiều người trong số các ứng dụng này sử dụng một giao lộ của datasets có sẵn. Phân tích ứng dụng có nghĩa vụ phải được thiết kế để cung cấp các kiến thức hữu dụng để tạo ra hoặc cải thiện giá trị. Chất lượng của thông tin phải được liên quan trực tiếp đến cách quy trình kinh doanh là hoặc là dự kiến sẽ được cải thiện bằng cách bỏ qua vấn đề dữ liệu dẫn đến tác động tiêu cực không mong muốn hoặc tốt hơn chất lượng dữ liệu, và có thể có các mức độ khác nhau của lãi suất trong khẳng định mức độ khả năng sử dụng và acceptability cho datasets mua lại bởi các bên khác nhau.Điều này có nghĩa là, cho phạm vi của dự án phân tích dữ liệu lớn khác nhau, bạn phải xác định những mong đợi tập thể người dùng bằng cách tham gia của người tiêu dùng khác nhau lớn dữ liệu Analytics để thảo luận về
đang được dịch, vui lòng đợi..
5 Data Governance for Big Data Analytics:
Considerations for Data Policies and Processes
It should not come as a surprise that in a big data environment, much like any environment, the end users might have concerns about the believability of analytical results. This is particularly true when there is limited visibility into trustworthiness of the data sources. One added challenge is that even if the producers of the data sources are known, the actual derivation of the acquired datasets may still remain opaque. Striving for data trustworthiness has driven the continued development and maturation of processes and tools for data quality assurance, data standardization, and data cleansing. In general, data quality is generally seen as a mature discipline, particularly when the focus is evaluating datasets and applying remedial or corrective actions to data to ensure that the datasets are fit for the purposes for which they were originally intended.
5.1 THE EVOLUTION OF DATA GOVERNANCE
In the past 5 years or so, there have been a number of realizations that have, to some extent, disrupted this perception of “data quality maturity,” namely:
• Correct versus correction: In many environments, tools are used to fix data, not to ensure that the data is valid or correct. What was once considered to be the cutting edge in terms of identifying and then fixing data errors has, to some extent, fallen out of favor in lieu of process-oriented validation, root cause analysis, and remediation.
• Data repurposing: More organizational stakeholders recognize that datasets created for one functional purpose within the enterprise (such as sales, marketing, accounts payable, or procurement to name a few) are used multiple times in different contexts, particularly for reporting and analysis. The implication is that data quality can no longer be measured in terms of “fitness for purpose,” but instead must be evaluated in terms of “fitness for purposes,” taking all downstream uses and quality requirements into account.
• The need for oversight: This realization, which might be considered a follow-on to the first, is that ensuring the usability of data for all purposes requires more comprehensive oversight. Such oversight should include monitored controls incorporated into the system development life cycle and across the application infrastructure.
These realizations lead to the discipline called data governance. Data governance describes the processes for defining corporate data policies, describing processes for operationalizing observance of those policies, along with the organizational structures that include data governance councils and data stewards put in place to monitor, and hopefully ensure compliance with those data policies.
Stated simply, the objective of data governance is to institute the right levels of control to achieve one of three outcomes:
1. Alert: Identify data issues that might have negative business impact.
2. Triage: Prioritize those issues in relation to their corresponding business value drivers.
3. Remediate: Have data stewards take the proper actions when alerted to the existence of those issues.
When focused internally, data governance not only enables a degree of control for data created and shared within an organization, it empowers the data stewards to take corrective action, either through communication with the original data owners or by direct data intervention (i.e., “correcting bad data”) when necessary.
5.2 BIG DATA AND DATA GOVERNANCE
Naturally, concomitant with the desire for measurably high quality information in a big data environment is the inclination to institute “big data governance.” It is naive, however, to assert that when it comes to big data governance one should adopt the traditional approaches to data quality. Furthermore, one cannot assume that just because vendors, system integrators, and consultants stake their claims over big data by stressing the need for “big data quality” that the same methods and tools can be used to monitor, review, and correct data streaming into a big data platform.
Upon examination, the key characteristics of big data analytics are not universally adaptable to the conventional approaches to data quality and data governance. For example, in a traditional approach to data quality, levels of data usability are measured based on the idea of “data quality dimensions,” such as:
• Accuracy, referring to the degree to which the data values are correct.
• Completeness, which specifies the data elements that must have values.
• Consistency of related data values across different data instances.
• Currency, which looks at the “freshness” of the data and whether the values are up to date or not.
• Uniqueness, which specifies that each real-world item is represented once and only once within the dataset.
These types of measures are generally intended to validate data using defined rules, catch any errors when the input does not conform to those rules, and correct recognized errors when the situations allow it. This approach typically targets moderately sized datasets, from known sources, with structured data, with a relatively small set of rules. Operational and analytical applications of limited size can integrate data quality controls, alerts, and corrections, and those corrections will reduce the downstream negative impacts.
5.3 THE DIFFERENCE WITH BIG DATASETS
On the other hand, big datasets neither exhibit these characteristics, nor do they have similar types of business impacts. Big data analytics is generally centered on consuming massive amounts of a combination of structured and unstructured data from both machine-generated and human sources. Much of the analysis is done without considering the business impacts of errors or inconsistencies across the different sources, from where the data originated, or how frequently it is acquired.
Big data applications look at many input streams originating from within and outside the organization, some taken from a variety of social networking streams, syndicated data streams, news feeds, preconfigured search filters, public or open-sourced datasets, sensor networks, or other unstructured data streams. Such diverse datasets resist singular approaches to governance.
When the acquired datasets and data streams originate outside the organization, there is little facility for control over the input. The original sources are often so obfuscated that there is little capacity to even know who created the data in the first place, let alone enable any type of oversight over data creation.
Another issue involves the development and execution model for big data applications. Data analysts are prone to develop their own models in their private sandbox environments. In these cases, the developers often bypass traditional IT and data management channels, opening greater possibilities for inconsistencies with sanctioned IT projects. This is complicated more as datasets are tapped into or downloaded directly without IT’s intervention.
Consistency (or the lack thereof) is probably the most difficult issue. When datasets are created internally and a downstream user recognizes a potential error, that issue can be communicated to the originating system’s owners. The owners then have the opportunity to find the root cause of the problems and then correct the processes that led to the errors.
But with big data systems that absorb massive volumes of data, some of which originates externally, there are limited opportunities to engage process owners to influence modifications or corrections to the source. On the other hand, if you opt to “correct” the recognized data error, you are introducing an inconsistency with the original source, which at worst can lead to incorrect conclusions and flawed decision making.
5.4 BIG DATA OVERSIGHT: FIVE KEY CONCEPTS
The conclusion is that the standard approach to data governance in which data policies defined by an internal governance council direct control of the usability of datasets cannot be universally applied to big data applications. And yet there is definitely a need for some type of oversight that can ensure that the datasets are usable and that the analytic results are trustworthy. One way to address the need for data quality and consistency is to leverage the concept of data policies based on the information quality characteristics that are important to the big data project.
This means considering the intended uses of the results of the analyses and how the inability to exercise any kind of control on the original sources of the information production flow can be mitigated by the users on the consumption side. This approach requires a number of key concepts for data practitioners and business process owners to keep in mind:
• managing consumer data expectations;
• identifying the critical data quality dimensions;
• monitoring consistency of metadata and reference data as a basis for entity extraction;
• repurposing and reinterpretation of data;
• data enrichment and enhancement when possible.
5.4.1 Managing Consumer Data Expectations
There may be a wide variety of users consuming the results of the spectrum of big data analytics applications. Many of these applications use an intersection of available datasets. Analytics applications are supposed to be designed to provide actionable knowledge to create or improve value. The quality of information must be directly related to the ways the business processes are either expected to be improved by better quality data or how ignoring data problems leads to undesired negative impacts, and there may be varied levels of interest in asserting levels of usability and acceptability for acquired datasets by different parties.
This means, for the scope of the different big data analytics projects, you must ascertain these collective user expectations by engaging the different consumers of big data analytics to discuss
Considerations for Data Policies and Processes
It should not come as a surprise that in a big data environment, much like any environment, the end users might have concerns about the believability of analytical results. This is particularly true when there is limited visibility into trustworthiness of the data sources. One added challenge is that even if the producers of the data sources are known, the actual derivation of the acquired datasets may still remain opaque. Striving for data trustworthiness has driven the continued development and maturation of processes and tools for data quality assurance, data standardization, and data cleansing. In general, data quality is generally seen as a mature discipline, particularly when the focus is evaluating datasets and applying remedial or corrective actions to data to ensure that the datasets are fit for the purposes for which they were originally intended.
5.1 THE EVOLUTION OF DATA GOVERNANCE
In the past 5 years or so, there have been a number of realizations that have, to some extent, disrupted this perception of “data quality maturity,” namely:
• Correct versus correction: In many environments, tools are used to fix data, not to ensure that the data is valid or correct. What was once considered to be the cutting edge in terms of identifying and then fixing data errors has, to some extent, fallen out of favor in lieu of process-oriented validation, root cause analysis, and remediation.
• Data repurposing: More organizational stakeholders recognize that datasets created for one functional purpose within the enterprise (such as sales, marketing, accounts payable, or procurement to name a few) are used multiple times in different contexts, particularly for reporting and analysis. The implication is that data quality can no longer be measured in terms of “fitness for purpose,” but instead must be evaluated in terms of “fitness for purposes,” taking all downstream uses and quality requirements into account.
• The need for oversight: This realization, which might be considered a follow-on to the first, is that ensuring the usability of data for all purposes requires more comprehensive oversight. Such oversight should include monitored controls incorporated into the system development life cycle and across the application infrastructure.
These realizations lead to the discipline called data governance. Data governance describes the processes for defining corporate data policies, describing processes for operationalizing observance of those policies, along with the organizational structures that include data governance councils and data stewards put in place to monitor, and hopefully ensure compliance with those data policies.
Stated simply, the objective of data governance is to institute the right levels of control to achieve one of three outcomes:
1. Alert: Identify data issues that might have negative business impact.
2. Triage: Prioritize those issues in relation to their corresponding business value drivers.
3. Remediate: Have data stewards take the proper actions when alerted to the existence of those issues.
When focused internally, data governance not only enables a degree of control for data created and shared within an organization, it empowers the data stewards to take corrective action, either through communication with the original data owners or by direct data intervention (i.e., “correcting bad data”) when necessary.
5.2 BIG DATA AND DATA GOVERNANCE
Naturally, concomitant with the desire for measurably high quality information in a big data environment is the inclination to institute “big data governance.” It is naive, however, to assert that when it comes to big data governance one should adopt the traditional approaches to data quality. Furthermore, one cannot assume that just because vendors, system integrators, and consultants stake their claims over big data by stressing the need for “big data quality” that the same methods and tools can be used to monitor, review, and correct data streaming into a big data platform.
Upon examination, the key characteristics of big data analytics are not universally adaptable to the conventional approaches to data quality and data governance. For example, in a traditional approach to data quality, levels of data usability are measured based on the idea of “data quality dimensions,” such as:
• Accuracy, referring to the degree to which the data values are correct.
• Completeness, which specifies the data elements that must have values.
• Consistency of related data values across different data instances.
• Currency, which looks at the “freshness” of the data and whether the values are up to date or not.
• Uniqueness, which specifies that each real-world item is represented once and only once within the dataset.
These types of measures are generally intended to validate data using defined rules, catch any errors when the input does not conform to those rules, and correct recognized errors when the situations allow it. This approach typically targets moderately sized datasets, from known sources, with structured data, with a relatively small set of rules. Operational and analytical applications of limited size can integrate data quality controls, alerts, and corrections, and those corrections will reduce the downstream negative impacts.
5.3 THE DIFFERENCE WITH BIG DATASETS
On the other hand, big datasets neither exhibit these characteristics, nor do they have similar types of business impacts. Big data analytics is generally centered on consuming massive amounts of a combination of structured and unstructured data from both machine-generated and human sources. Much of the analysis is done without considering the business impacts of errors or inconsistencies across the different sources, from where the data originated, or how frequently it is acquired.
Big data applications look at many input streams originating from within and outside the organization, some taken from a variety of social networking streams, syndicated data streams, news feeds, preconfigured search filters, public or open-sourced datasets, sensor networks, or other unstructured data streams. Such diverse datasets resist singular approaches to governance.
When the acquired datasets and data streams originate outside the organization, there is little facility for control over the input. The original sources are often so obfuscated that there is little capacity to even know who created the data in the first place, let alone enable any type of oversight over data creation.
Another issue involves the development and execution model for big data applications. Data analysts are prone to develop their own models in their private sandbox environments. In these cases, the developers often bypass traditional IT and data management channels, opening greater possibilities for inconsistencies with sanctioned IT projects. This is complicated more as datasets are tapped into or downloaded directly without IT’s intervention.
Consistency (or the lack thereof) is probably the most difficult issue. When datasets are created internally and a downstream user recognizes a potential error, that issue can be communicated to the originating system’s owners. The owners then have the opportunity to find the root cause of the problems and then correct the processes that led to the errors.
But with big data systems that absorb massive volumes of data, some of which originates externally, there are limited opportunities to engage process owners to influence modifications or corrections to the source. On the other hand, if you opt to “correct” the recognized data error, you are introducing an inconsistency with the original source, which at worst can lead to incorrect conclusions and flawed decision making.
5.4 BIG DATA OVERSIGHT: FIVE KEY CONCEPTS
The conclusion is that the standard approach to data governance in which data policies defined by an internal governance council direct control of the usability of datasets cannot be universally applied to big data applications. And yet there is definitely a need for some type of oversight that can ensure that the datasets are usable and that the analytic results are trustworthy. One way to address the need for data quality and consistency is to leverage the concept of data policies based on the information quality characteristics that are important to the big data project.
This means considering the intended uses of the results of the analyses and how the inability to exercise any kind of control on the original sources of the information production flow can be mitigated by the users on the consumption side. This approach requires a number of key concepts for data practitioners and business process owners to keep in mind:
• managing consumer data expectations;
• identifying the critical data quality dimensions;
• monitoring consistency of metadata and reference data as a basis for entity extraction;
• repurposing and reinterpretation of data;
• data enrichment and enhancement when possible.
5.4.1 Managing Consumer Data Expectations
There may be a wide variety of users consuming the results of the spectrum of big data analytics applications. Many of these applications use an intersection of available datasets. Analytics applications are supposed to be designed to provide actionable knowledge to create or improve value. The quality of information must be directly related to the ways the business processes are either expected to be improved by better quality data or how ignoring data problems leads to undesired negative impacts, and there may be varied levels of interest in asserting levels of usability and acceptability for acquired datasets by different parties.
This means, for the scope of the different big data analytics projects, you must ascertain these collective user expectations by engaging the different consumers of big data analytics to discuss
đang được dịch, vui lòng đợi..
Các ngôn ngữ khác
Hỗ trợ công cụ dịch thuật: Albania, Amharic, Anh, Armenia, Azerbaijan, Ba Lan, Ba Tư, Bantu, Basque, Belarus, Bengal, Bosnia, Bulgaria, Bồ Đào Nha, Catalan, Cebuano, Chichewa, Corsi, Creole (Haiti), Croatia, Do Thái, Estonia, Filipino, Frisia, Gael Scotland, Galicia, George, Gujarat, Hausa, Hawaii, Hindi, Hmong, Hungary, Hy Lạp, Hà Lan, Hà Lan (Nam Phi), Hàn, Iceland, Igbo, Ireland, Java, Kannada, Kazakh, Khmer, Kinyarwanda, Klingon, Kurd, Kyrgyz, Latinh, Latvia, Litva, Luxembourg, Lào, Macedonia, Malagasy, Malayalam, Malta, Maori, Marathi, Myanmar, Mã Lai, Mông Cổ, Na Uy, Nepal, Nga, Nhật, Odia (Oriya), Pashto, Pháp, Phát hiện ngôn ngữ, Phần Lan, Punjab, Quốc tế ngữ, Rumani, Samoa, Serbia, Sesotho, Shona, Sindhi, Sinhala, Slovak, Slovenia, Somali, Sunda, Swahili, Séc, Tajik, Tamil, Tatar, Telugu, Thái, Thổ Nhĩ Kỳ, Thụy Điển, Tiếng Indonesia, Tiếng Ý, Trung, Trung (Phồn thể), Turkmen, Tây Ban Nha, Ukraina, Urdu, Uyghur, Uzbek, Việt, Xứ Wales, Yiddish, Yoruba, Zulu, Đan Mạch, Đức, Ả Rập, dịch ngôn ngữ.
- The movie star who bought the Learjet co
- As a result, more of these tourists migh
- Either the letter or the full price fiel
- hard
- hate those guys to take your friend's fa
- Love, Chunibyo & Other Delusions (中二病でも恋
- If multiple bands are detected under str
- current
- affect your emotions?
- ยินดี
- tôi sẻ nói làm sao để giử dáng tốt
- versus
- hate those guys to take your friend's fa
- Là một thành phố nằm ở phía Tây Bắc nước
- Socio-economic background. The previous
- Tôi muốn bỏ mọi thứ trong cuộc sống hiện
- violent
- very hard
- ForProblem19.27,determinetheprobability(
- hoa đào
- There is an established institutional me
- The choice of manufacturing process is n
- English News
- The choice of manufacturing process is n
