R/04-harmonized_data_summarize.R
harmonized_dossier_summarize.RdAssesses and summarizes the content and structure of a harmonized dossier and generates reports of the results. This function can be used to evaluate data structure, presence of specific fields, coherence across elements, and data dictionary formats, and to summarize additional information about variable distributions and descriptive statistics.
harmonized_dossier_summarize(
harmonized_dossier,
group_by = attributes(harmonized_dossier)$`Rmonize::harmonized_col_dataset`,
dataschema = attributes(harmonized_dossier)$`Rmonize::DataSchema`,
data_proc_elem = attributes(harmonized_dossier)$`Rmonize::Data Processing Element`,
taxonomy = NULL,
valueType_guess = FALSE
)A list containing the harmonized dataset(s).
A character string identifying the column in the dataset to use as a grouping variable. Elements will be grouped by this column.
A DataSchema object.
A Data Processing Elements object.
An optional data frame identifying a variable classification schema.
Whether the output should include a more accurate valueType that could be applied to the dataset. FALSE by default.
A list of data frames containing overall assessment reports and summaries grouped by harmonized dataset.
A harmonized dossier is a named list containing one or more data frames, which are harmonized datasets. A harmonized dossier is generally the product of applying processing to a dossier object The name of each harmonized dataset (data frame) is taken from the reference input dataset. A harmonized dossier also contains the DataSchema and Data Processing Elements used in processing as attributes.
A DataSchema is the list of core variables to generate across datasets and
related metadata. A DataSchema object is a list of data frames with elements
named 'Variables' (required) and 'Categories' (if any). The 'Variables'
element must contain at least the name column, and the 'Categories'
element must contain at least the variable and name columns to be usable
in any function. In 'Variables' the name column must also have unique
entries, and in 'Categories' the combination of variable and name columns
must also be unique.
The Data Processing Elements specifies the algorithms used to process input
variables into harmonized variables in the DataSchema format. It is also
contains metadata used to generate documentation of the processing.
A Data Processing Elements object is a data frame with specific columns
used in data processing: dataschema_variable, input_dataset,
input_variables, Mlstr_harmo::rule_category and Mlstr_harmo::algorithm.
To initiate processing, the first entry must be the creation of a harmonized
primary identifier variable (e.g., participant unique ID).
A taxonomy is a classification schema that can be defined for variable
attributes. A taxonomy is usually extracted from an
Opal environment, and a
taxonomy object is a data frame that must contain at least the columns
taxonomy, vocabulary, and terms. Additional details about Opal
taxonomies are
available online.
The valueType is a declared property of a variable that is required in certain functions to determine handling of the variables. Specifically, valueType refers to the OBiBa data type of a variable. The valueType is specified in a data dictionary in a column 'valueType' and can be associated with variables as attributes. Acceptable valueTypes include 'text', 'integer', 'decimal', 'boolean', datetime', 'date'. The full list of OBiBa valueType possibilities and their correspondence with R data types are available using valueType_list. The valueType can be used to coerce the variable to the corresponding data type.
{
harmonized_dossier <- Rmonize_DEMO$harmonized_dossier
# summary harmonization
harmonized_dossier_summarize(harmonized_dossier)
}
#> Warning:
#> An additional variable `Rmonize::harmonized_col_dataset` has been created
#> with harmonized dataset names as values for each harmonized dataset.
#>
#> Useful tip:
#> To avoid this message, we recommend to set `harmonized_col_dataset` as a
#> categorical variable DataSchema, or to set `add_col_dataset` = FALSE.
#> - DATA DICTIONARY ASSESSMENT: data_dict --------------
#> Assess the standard adequacy of naming
#> Assess the uniqueness of variable names
#> Assess the presence of possible duplicated columns
#> Assess the presence of empty rows in the data dictionary
#> Assess the presence of empty columns in the data dictionary
#> Assess the completion of `label(:xx)` column in 'Variables'
#> Assess the `valueType` column in 'Variables'
#> Generate report
#>
#> - WARNING MESSAGES (if any): --------------------------------------------
#>
#> - DATASET ASSESSMENT: pooled_harmonized_dataset (empty dataset) --------------------------
#> Assess the standard adequacy of naming
#> Assess the presence of variable names both in dataset and data dictionary
#> Assess the presence of possible duplicated variable in the dataset
#> Assess the presence of duplicated participants in the dataset
#> Assess the presence of empty rows in the data dictionary
#> Assess the presence all NA(s) of columns in the data dictionary
#> Assess the presence of categories not in the data dictionary
#> Assess the `valueType` comparison in dataset and data dictionary
#> Generate report
#>
#> - WARNING MESSAGES (if any): -------------------------------------------------
#>
#> - DATASET SUMMARIZE: pooled_harmonized_dataset (empty dataset) --------------------------
#> Summarize the data type of each variable across the dataset
#> Summarize global information (Overview)
#> Generate report
#> $`Harmonization Overview`
#> # A tibble: 15 × 2
#> `Quality control of dataset` `(all)`
#> <chr> <chr>
#> 1 "Date" "2023-12-20"
#> 2 "Name of the dataset" "pooled_harmonized_da…
#> 3 " Identifier Variable" ""
#> 4 " Variables" " "
#> 5 " Total number of variables (incl. identifier)" "2"
#> 6 " Total number of empty columns" "2"
#> 7 " Data type in dictionary (valueType)" " "
#> 8 " Nb. text variables" "2"
#> 9 " Nb. date variables" "0"
#> 10 " Nb. datetime variables" "0"
#> 11 " Nb. numerical variables" "0"
#> 12 " Nb. categorical variables" "0"
#> 13 " Rows" "(all)"
#> 14 " Total number of observations" "0"
#> 15 " Nb. distinct observations" "0"
#>
#> $`Harmonized Data dictionary assessement`
#> # A tibble: 1 × 5
#> sheet col_name name_var Quality assessment c…¹ value
#> <chr> <chr> <chr> <chr> <chr>
#> 1 Variables name Rmonize::harmonized_col_datas… [INFO] - Incompatible… NA
#> # ℹ abbreviated name: ¹`Quality assessment comment`
#>
#> $`Harmonized Dataset assessment`
#> # A tibble: 1 × 5
#> `index in data dict.` name Quality assessment c…¹ value suggestion
#> <int> <chr> <chr> <chr> <chr>
#> 1 2 Rmonize::harmon… [INFO] - Incompatible… NA NA
#> # ℹ abbreviated name: ¹`Quality assessment comment`
#>
#> $`Harmonized Variables summary (all)`
#> # A tibble: 2 × 7
#> `index in data dict.` name `label:en` Data Dictionary valu…¹
#> <int> <chr> <chr> <chr>
#> 1 1 adm_unique_id Unique id… text
#> 2 2 Rmonize::harmonized_c… Rmonize::… text
#> # ℹ abbreviated name: ¹`Data Dictionary valueType`
#> # ℹ 3 more variables: `Estimated dataset valueType` <chr>,
#> # `Actual dataset valueType` <chr>, `Categories in data dictionary` <chr>
#>
#> $`Text variable summary`
#> # A tibble: 2 × 7
#> `index in data dict.` name `label:en` Data Dictionary valu…¹
#> <int> <chr> <chr> <chr>
#> 1 1 adm_unique_id Unique id… text
#> 2 2 Rmonize::harmonized_c… Rmonize::… text
#> # ℹ abbreviated name: ¹`Data Dictionary valueType`
#> # ℹ 3 more variables: `Estimated dataset valueType` <chr>,
#> # `Actual dataset valueType` <chr>, `Categories in data dictionary` <chr>
#>