Commission Implementing Regulation (EU) 2021/2224 of 16 November 2021 laying down the details of the automated data quality control mechanisms and procedures, the common data quality indicators and the minimum quality standards for storage of data, pursuant to Article 37(4) of Regulation (EU) 2019/818 of the European Parliament and of the Council

• CELEX: Implementing Regulation (EU) 2021/2224
• Articles: 8

Article 1 Scope and subject matter

1.   This Regulation lays down the details of the automated data quality control mechanisms and procedures on the data stored in the EU information systems and interoperability components referred to in Article 37(1) of Regulation (EU) 2019/818.

2.   This Regulation also lays down details concerning the common data quality indicators and the minimum quality standards for storage of data, in particular regarding biometric data, in the EU information systems and interoperability components referred to in Article 37(1) of Regulation (EU) 2019/818.

3.   The measures referred to in paragraphs 2 and 3 are without prejudice to any specific provisions concerning data quality related to the EU information systems laid down in Union law.

4.   This Regulation applies to the EU information systems and interoperability components referred to in Article 37(1) of Regulation (EU) 2019/818.

Article 2 Definitions

For the purposes of this Regulation, the following definitions apply:

(a)

‘input data’ means data, subject to data quality checks for the purpose of being stored in a EU information system or interoperability component referred to in Article 37(1) of Regulation (EU) 2019/818;

(b)

‘data cleaning mechanism’ means a mechanism carrying out checks in order to ensure the scheduled erasure of data stored in a EU information system or interoperability component in accordance with Union law;

(c)

‘issue detection mechanism’ means a mechanism carrying out checks in order to identify data that does not meet the data quality rules or standards;

(d)

‘blocking rules’ means rules or a set of rules that measure the degree to which input data is compliant with defined data requirements conditioning their storage or use or both. They also include data quality rules governing each EU information system that must be complied with before data can be entered in the system. Input data not complying with a blocking rule will be rejected from being entered and stored in the EU information system and interoperability component;

(e)

‘soft rules’ means rules or a set of rules that measure the degree to which the input data is compliant with the defined data requirements conditioning its relevance or optimal use or both. Soft rules shall not prevent the entry and storage of non-compliant input data. They also include data quality rules governing each EU information system that should be complied with before data can be entered in the system. Input data not complying with a soft rule shall be entered into the EU information system or interoperability component with a data quality issue flag, notification or warning message.

Article 3 Automated data quality control mechanisms and procedures

1.   Automated quality checks shall be conducted on the data entered and stored in the shared biometric matching service and the common identity repository in accordance with the rules laid down in Article 4.

2.   Automated quality checks shall be conducted on the data entered and stored in the EU information systems referred to in Article 37(1) of Regulation (EU) 2019/818 in accordance with the rules governing data quality control mechanisms in those systems.

3.   Data quality control mechanisms shall be triggered in accordance with the rules governing data quality in those EU information systems and interoperability components.

4.   In order to determine compliance of input data with the blocking or soft rules applying to them, the data quality control mechanisms referred to in paragraph 3 shall comply with Section 1 of the Annex to this Regulation.

5.   If the input data is to be entered into the EU information system or interoperability components referred to in Article 37(1) of Regulation (EU) 2019/818, the data quality control mechanisms shall assess the extent to which the data complies with each data quality indicator by applying the data quality standard of each indicator. As a result of that assessment, the data quality control mechanisms shall assign to the input data a data quality classification pursuant to the process laid down in Sections 2 and 3 of the Annex to this Regulation.

6.   The common data quality indicators shall be the following: completeness, accuracy, timeliness, uniqueness and consistency.

7.   eu-LISA shall implement the data quality standards for each indicator pursuant to the procedures laid down in Article 5.

8.   Data cleaning and issue detection mechanisms shall regularly check the validity and the data quality compliance of the data stored in the EU information systems and interoperability components referred to in Article 37(1) of Regulation (EU) 2019/818, in accordance with Section 4 of the Annex to this Regulation.

Article 4 Automated data quality control mechanisms for data entered and stored

1.   To enhance data quality, automated data quality control mechanisms shall be set up to support the entering and storage of data complying with data quality requirements in the EU information systems and interoperability components referred to in Article 37(1) of Regulation (EU) 2019/818. The data shall be entered and stored in accordance with the rules governing data quality in those EU information systems or interoperability components.

2.   For the purpose of the entry of data by the duly authorised staff in the EU information systems and interoperability components referred to in Article 37(1) of Regulation (EU) 2019/818, the automated quality control mechanisms shall verify the common quality indicators referred to in Section 2 of the Annex to this Regulation.

3.   Data quality control mechanisms shall allow the application of blocking rules and soft rules in accordance with Article 3(4) governing data quality in the EU Information Systems and interoperability components referred to in Article 37(1) of Regulation (EU) 2019/818.

Article 5 Procedures governing the data quality control indicators, standards and mechanisms

1.   eu-LISA shall be responsible for ensuring that the data quality rules are appropriate for achieving the objectives of the EU information systems and interoperability components referred to in Article 37(1) of Regulation (EU) 2019/818.

2.   eu-LISA shall also be responsible for implementing the minimum quality standards for storage of biometric data in the EU information systems and interoperability components referred to in Article 37(1) of Regulation (EU) 2019/818.

3.   For the purposes of paragraphs 1 and 2 of this Article, eu-LISA shall take into consideration the specific needs of each EU information system and interoperability component referred to in Article 37(1) of Regulation (EU) 2019/818 and shall be assisted and advised by experts from the Commission, the Member States and the Union agencies using those EU information systems and interoperability components.

4.   The following data quality control procedures shall apply:

(a)

eu-LISA shall implement the data quality standards for each data quality indicator in accordance with Section 2 of the Annex to this Regulation;

(b)

values shall be assigned to the standards separately for each of the data quality indicators referred to in Article 3(6). For each indicator, the standards may assign different values depending on the category of data;

(c)

when so required and after the issuance of the regular data quality reports in accordance with Article 6, eu-LISA shall assess the appropriateness of the values and the standards and amend them to the extent to which they are no longer appropriate;

(d)

when so required and after the issuance of the regular data quality reports in accordance with Article 6, eu-LISA shall assess the appropriateness of the data quality control mechanisms allowing for the application of blocking and soft rules and amend them where relevant;

(e)

with a view of amending the data quality standards and their values, and to taking any decision on the data quality control mechanisms pursuant to points (c) and (d) of this paragraph, eu-LISA shall consult the Advisory Groups for each of the EU information systems and interoperability components referred to in Article 37(1) of Regulation (EU) 2019/818.

5.   eu-LISA shall base itself on the assessment of the functioning of the data quality control mechanisms with a view to amending the values of the data quality standards and to taking any decision on the data quality control mechanisms.

Article 6 Reports on automated data quality control mechanisms and procedures and common data quality indicators pursuant to Article 37(3) of Regulation (EU) 2019/818

1.   Reports referred to in Article 37(3) of Regulation (EU) 2019/818 shall be produced by the central repository for reporting and statistics pursuant to Article 39(1) of Regulation (EU) 2019/818 and the rules laid down in the Delegated Regulation 2021/2222.

2.   Reports shall not include personal data and contain at least the following information metrics on the quality of data, extracted via a tool or set of tools developed for these purposes:

(a)

for alphanumeric and biometric data assessed against blocking rules and soft rules, compliance with data quality indicators:

(1)

completeness (%);

(2)

accuracy (%):

(3)

uniqueness (%);

(4)

timeliness (%);

(5)

consistency of data (%);

(b)

completeness of application files (%);

(c)

compliance of data with the ‘good quality’ classification (%);

(d)

compliance of data with the ‘low quality’ classification (%);

(e)

results of the data cleaning mechanism;

(f)

results of the issue detection mechanism;

(g)

data fields that cause frequent quality issues;

(h)

top 10 issues list for each of the categories referred to in points (a) to (g);

(i)

the Member States concerned by the top 10 quality issues.

3.   eu-LISA shall develop a central monitoring capacity for data quality and for producing the reports referred to in this Article on a weekly basis.

Article 7 Entry into force

This Regulation shall enter into force on the twentieth day following that of its publication in the Official Journal of the European Union .

Schedules & Appendices

ANNEX

ANNEX

SECTION 1

Automated data quality control mechanism for data to be entered

Data entered into the EU information systems and interoperability components will be subject to automated data quality control mechanisms based on blocking and soft rules as defined in Article 2. These are the rules established in the EU information systems and interoperability components which determine whether the entry and storage of input data shall be allowed or rejected. The blocking and soft rules are established based on the following parameters: length, format, type, conformity to quality standards, semantics, repetition and syntax.

SECTION 2

General considerations on the common data quality indicators and minimum quality standards for data to be entered

The input data subject to the quality verification process shall be assessed against the data quality rules defined in each EU information system and interoperability component, as set out in Section 1. If the rules applying to the input data do not prevent entry and storage, the data quality control mechanisms shall measure the quality of the input data by using the data quality indicators applying to them.

The data quality control mechanisms shall measure the quality of input data according to each relevant indicator. The data quality control mechanisms shall take into account a weighing coefficient to calculate the relative weight of each indicator on the overall quality of the input data.

To that effect, the data quality control mechanisms shall be adapted to apply to a single collection of data within a record, or to a database.

After applying the weighing coefficient to the input data, the data quality control mechanisms shall produce an input data profile containing the results of the application of the indicator standards, for example numerical values evaluating the quality of the input data under each indicator.

Table 1 lists the minimum set of data quality indicators, for example indicators that shall always apply to input data, in accordance with the rules applied by each EU information system and interoperability component. Such indicators are the following: completeness, accuracy, consistency, timeliness, uniqueness.

Table 1

List of minimum data quality indicators

Indicator

Description

Main scope of applicability

Unit of measurement

Completeness

Means the degree to which the input data has values for all the expected attributes and related requirements in a specific context of use. Measures whether all the mandatory data is provided and the database (or sectoral) listings meet the set demands.

Mandatory data fields (alphanumeric and biometric)

Data completeness rate: ratio of the number of data cells provided to the number of data cells required

Accuracy

Means the degree to which the input data represents closeness of estimates to the unknown true values. It can be either or both among data regarding one entity and across similar data for comparable entities

Alphanumeric and biometric data

Sampling error rates, unit non-response rate, item non-response rate, data capture error rates, etc.

Consistency

Means the degree to which the input data has attributes that are free from contradiction and are coherent with other data in a specific context of use. Measures the degree to which a set of data satisfies defined business rules applying to those data across them, means the absence of a conflict of data content. It can be either or both among data regarding one entity and across similar data for comparable entities.

Alphanumeric data

Percentage

Timeliness

Means the degree to which the input data is provided within a predefined date or time that condition the validity of the data or its context of use. Measures how up-to-date the data is, and whether the data required can be provided by the required time.

Alphanumeric and biometric data

Time lag -final: number of days from the last day of the reference to the day the input data is provided

Uniqueness

Means the degree to which the input data is not duplicated in the same EU information system or interoperability component.

Mandatory data fields (alphanumeric and biometric)

Percentage of data units which are not duplicated

The accuracy indicator for biometric data also includes resolution. Resolution measures the degree to which the input data contains the required amount of points or pixels by unit of length. Unit to display on screen pixel: pi unit for printing; dot pi for output systems. Pixel one or several bits (range of colours ex: 16 colours 4b, 256 8b, 16b 65k, 24b 16.5mio)

SECTION 3

Data Quality Classification

After the development of the input data profile, referred to in Section 2, the input data shall be assigned with a data quality classification. The following data quality classifications shall apply:

(a)

‘good quality’ means the input data profile demonstrates the required compliance with the applicable data quality indicator;

(b)

‘low quality’ means the input data profile does not demonstrate the required compliance with the applicable data quality indicators, in the case of a soft rule;

(c)

‘rejected’ means the input data profile does not demonstrate the required compliance with the applicable data quality indicators in case of a blocking rule.

Where the input data is assigned with a ‘good quality’ classification, the data shall be stored into the system or component without any data quality alert.

Where the input data is assigned with a ‘low quality’ classification, the data shall be stored into the system or component and with a data quality alert. An alert shall indicate that the input data shall be rectified and the reason why the input data does not demonstrate the required compliance with the applicable data quality indicators. Where possible, the alert shall identify the data field(s) or the data content(s) or both affected by data quality issues and suggest the changes necessary for the input data to meet the ‘good quality’ classification.

SECTION 4

Data Quality Monitoring

Two types of mechanisms shall be used for the purposes of Article 3(8):

(a)

Data cleaning mechanisms. Such mechanisms shall carry out checks to identify data for which the remaining retention period is less than the time defined in the legislation governing the relevant EU information system or interoperability component. Data cleaning mechanisms shall inform the Member State of the scheduled erasure of the data and allow them to adopt, if necessary, the appropriate measures.

(b)

Issue detection mechanisms. Such mechanisms shall carry out checks to identify the data that no longer meet one or more data quality rules or standards related to data quality indicators. Such checks may return an alert or notification to the responsible authority of the Member State indicating the reason why the data no longer meets one or more data quality rules or standards. Where possible, the alert shall suggest the changes necessary for the input data to meet the new rules or standards. In no case the application of such checks shall lead to automated deletion of data stored in the EU information systems or interoperability components. When new data is being entered into an EU information system or interoperability component while the issue detection mechanisms are running, the issue detection mechanisms shall not apply to those data.

eu-LISA may decide that ad hoc issue detection mechanisms shall be carried out in the EU information system and in the interoperability components upon the revision of the data quality rules or standards.

eu-LISA may consult the Advisory Group for one of the EU information systems or the interoperability components on whether to carry out an ad hoc issue detection mechanisms in its corresponding EU information system or interoperability component to the extent necessary for the purposes of that EU information system or interoperability component.