R/04-harmonized_data_summarize.R
harmonized_dossier_summarize.RdAssesses and summarizes the content and structure of a harmonized dossier and generates reports of the results. This function can be used to evaluate data structure, presence of specific fields, coherence across elements, and data dictionary formats, and to summarize additional information about variable distributions and descriptive statistics.
harmonized_dossier_summarize(harmonized_dossier)A list containing the harmonized dataset(s).
A list of data frames containing overall assessment reports and summaries grouped by harmonized dataset.
A harmonized dossier is a named list containing one or more data frames, which are harmonized datasets. A harmonized dossier is generally the product of applying processing to a dossier object. The name of each harmonized dataset (data frame) is taken from the reference input dataset. A harmonized dossier also contains the DataSchema and Data Processing Elements used in processing as attributes.
A DataSchema is the list of core variables to generate across datasets and
related metadata. A DataSchema object is a list of data frames with elements
named 'Variables' (required) and 'Categories' (if any). The 'Variables'
element must contain at least the name column, and the 'Categories'
element must contain at least the variable and name columns to be usable
in any function. In 'Variables' the name column must also have unique
entries, and in 'Categories' the combination of variable and name columns
must also be unique.
The Data Processing Elements specifies the input elements and processing algorithms
to generate harmonized variables in the DataSchema formats. It is also
contains metadata used to generate documentation of the processing.
A Data Processing Elements object is a data frame with specific columns
used in data processing: dataschema_variable, input_dataset,
input_variables, Mlstr_harmo::rule_category and Mlstr_harmo::algorithm.
To initiate processing, the first entry must be the creation of a harmonized
primary identifier variable (e.g., participant unique ID).
A taxonomy is a classification schema that can be defined for variable
attributes. A taxonomy is usually extracted from an
Opal environment, and a
taxonomy object is a data frame that must contain at least the columns
taxonomy, vocabulary, and terms. Additional details about Opal
taxonomies are
available online.
The valueType is a declared property of a variable that is required in certain functions to determine handling of the variables. Specifically, valueType refers to the OBiBa data type of a variable. The valueType is specified in a data dictionary in a column 'valueType' and can be associated with variables as attributes. Acceptable valueTypes include 'text', 'integer', 'decimal', 'boolean', datetime', 'date'. The full list of OBiBa valueType possibilities and their correspondence with R data types are available using valueType_list. The valueType can be used to coerce the variable to the corresponding data type.
{
# Use Rmonize_examples to run examples.
library(dplyr)
# Perform data processing
harmonized_dossier <- Rmonize_examples$`harmonized_dossier`
summary_harmo <- harmonized_dossier_summarize(harmonized_dossier)
glimpse(summary_harmo)
}
#> - DATA DICTIONARY ASSESSMENT: data_dict --------------
#> Assess the standard adequacy of naming
#> Assess the uniqueness and presence of variable names
#> Assess the presence of possible duplicated columns
#> Assess the presence of duplicated rows
#> Assess the presence of empty rows in the data dictionary
#> Assess the presence of empty column in the data dictionary
#> Assess the presence of categories not in the data dictionary
#> Assess the `valueType` column in 'Variables'
#> Assess the completion of `label` column in 'Variables'
#> Assess presence and completion of `label` column in 'Categories'
#> Assess the logical values of missing column in Categories
#> Generate report
#>
#> The data dictionary contains no errors/warnings.
#>
#> - WARNING MESSAGES (if any): --------------------------------------------
#>
#> - DATASET ASSESSMENT: pooled_harmonized_dataset --------------------------
#> Assess the standard adequacy of naming
#> Assess the presence of variable names both in dataset and data dictionary
#> Assess the presence of possible duplicated variable in the dataset
#> Assess the presence of possible duplicated participants
#> Assess the presence of unique value columns in dataset
#> Assess the presence of empty rows in the dataset
#> Assess the presence all empty variable in the dataset
#> Assess the Categories comparison in dataset and data dictionary
#> Assess the `valueType` comparison in dataset and data dictionary
#> Generate report
#>
#> - WARNING MESSAGES (if any): -------------------------------------------------
#>
#> - DATASET SUMMARIZE: pooled_harmonized_dataset --------------------------
#> Summarize the data type of each variable across the dataset
#> Summarize information for all variables
#> Summarize information for numerical variables
#> Summarize information for categorical variables
#> Summarize global information (Overview)
#> Generate report
#> List of 5
#> $ Harmonization overview : tibble [13 × 7] (S3: tbl_df/tbl/data.frame)
#> ..$ Harmonization overview: chr [1:13] "Date report generated" "Name of the dataset" " Harmonized identifier variable" " Harmonized dataset variable" ...
#> ..$ (all) : chr [1:13] "2025-06-13" "pooled_harmonized_dataset" "adm_unique_id" "adm_study_id" ...
#> ..$ Study 1 : chr [1:13] " " " " " " " " ...
#> ..$ Study 2 : chr [1:13] " " " " " " " " ...
#> ..$ Study 3 : chr [1:13] " " " " " " " " ...
#> ..$ Study 4 : chr [1:13] " " " " " " " " ...
#> ..$ Study 5 : chr [1:13] " " " " " " " " ...
#> $ Variables summary (all) : tibble [41 × 20] (S3: tbl_df/tbl/data.frame)
#> ..$ Index : chr [1:41] "1" "1" "1" "1" ...
#> ..$ Grouping variable: adm_study_id: chr [1:41] "[1] Study 1" "[2] Study 2" "[3] Study 3" "[4] Study 4" ...
#> ..$ Variable name : chr [1:41] "adm_unique_id" "adm_unique_id" "adm_unique_id" "adm_unique_id" ...
#> ..$ Variable label : chr [1:41] "Participant identifier" "Participant identifier" "Participant identifier" "Participant identifier" ...
#> ..$ Harmonization status : chr [1:41] "complete" "complete" "complete" "complete" ...
#> ..$ Quality assessment comment : chr [1:41] "[INFO] - Harmonized identifier variable." "[INFO] - Harmonized identifier variable." "[INFO] - Harmonized identifier variable." "[INFO] - Harmonized identifier variable." ...
#> ..$ Data dictionary valueType : chr [1:41] "text" "text" "text" "text" ...
#> ..$ Dataset valueType : chr [1:41] "text" "text" "text" "text" ...
#> ..$ Suggested valueType : chr [1:41] NA NA NA NA ...
#> ..$ Categorical variable : chr [1:41] "no" "no" "no" "no" ...
#> ..$ Categories in data dictionary : chr [1:41] NA NA NA NA ...
#> ..$ Non-valid categories : chr [1:41] NA NA NA NA ...
#> ..$ Number of rows : chr [1:41] "4000" "3000" "1000" "2500" ...
#> ..$ Number of valid values : chr [1:41] "4000" "3000" "1000" "2500" ...
#> ..$ Number of non-valid values : chr [1:41] "0" "0" "0" "0" ...
#> ..$ Number of empty values : chr [1:41] "0" "0" "0" "0" ...
#> ..$ % Valid values : chr [1:41] "100" "100" "100" "100" ...
#> ..$ % Non-valid values : chr [1:41] "0" "0" "0" "0" ...
#> ..$ % Empty values : chr [1:41] "0" "0" "0" "0" ...
#> ..$ Number of distinct values : chr [1:41] "4000" "3000" "1000" "2500" ...
#> $ Numerical variable summary : tibble [15 × 23] (S3: tbl_df/tbl/data.frame)
#> ..$ Index : chr [1:15] "3" "3" "3" "3" ...
#> ..$ Grouping variable: adm_study_id: chr [1:15] "[1] Study 1" "[2] Study 2" "[3] Study 3" "[4] Study 4" ...
#> ..$ Variable name : chr [1:15] "sdc_age_m" "sdc_age_m" "sdc_age_m" "sdc_age_m" ...
#> ..$ Variable label : chr [1:15] "Age of mother at first visit" "Age of mother at first visit" "Age of mother at first visit" "Age of mother at first visit" ...
#> ..$ Harmonization status : chr [1:15] "complete" "complete" "complete" "complete" ...
#> ..$ Quality assessment comment : chr [1:15] NA NA NA NA ...
#> ..$ Data dictionary valueType : chr [1:15] "integer" "integer" "integer" "integer" ...
#> ..$ Categorical variable : chr [1:15] "no" "no" "no" "no" ...
#> ..$ Number of rows : chr [1:15] "4000" "3000" "1000" "2500" ...
#> ..$ Number of valid values : chr [1:15] "3912" "2943" "1000" "2402" ...
#> ..$ Number of non-valid values : chr [1:15] "0" "0" "0" "0" ...
#> ..$ Number of empty values : chr [1:15] "88" "57" "0" "98" ...
#> ..$ % Valid values : chr [1:15] "97.8" "98.1" "100" "96.08" ...
#> ..$ % Non-valid values : chr [1:15] "0" "0" "0" "0" ...
#> ..$ % Empty values : chr [1:15] "2.2" "1.9" "0" "3.92" ...
#> ..$ Number of distinct values : chr [1:15] "27" "29" "31" "27" ...
#> ..$ Minimum : chr [1:15] "19" "16" "16" "20" ...
#> ..$ 1st quartile : chr [1:15] "28" "29" "28" "31" ...
#> ..$ Median : chr [1:15] "31" "32" "31" "34" ...
#> ..$ 3rd quartile : chr [1:15] "34" "34" "35" "36" ...
#> ..$ Maximum : chr [1:15] "47" "44" "47" "51" ...
#> ..$ Mean : chr [1:15] "30.83" "31.75" "31.29" "33.87" ...
#> ..$ Standard deviation : chr [1:15] "4.42" "4.25" "5.22" "3.93" ...
#> $ Categorical variable summary: tibble [21 × 21] (S3: tbl_df/tbl/data.frame)
#> ..$ Index : chr [1:21] "2" "4" "4" "4" ...
#> ..$ Grouping variable: adm_study_id : chr [1:21] "(all)" "[1] Study 1" "[2] Study 2" "[3] Study 3" ...
#> ..$ Variable name : chr [1:21] "adm_study_id" "sdc_marital_m" "sdc_marital_m" "sdc_marital_m" ...
#> ..$ Variable label : chr [1:21] "Study ID" "Marital status of mother at first visit" "Marital status of mother at first visit" "Marital status of mother at first visit" ...
#> ..$ Harmonization status : chr [1:21] "complete" "complete" "complete" "impossible" ...
#> ..$ Quality assessment comment : chr [1:21] "[INFO] - Harmonized dataset identifier variable." NA NA "[INFO] - Empty variable." ...
#> ..$ Data dictionary valueType : chr [1:21] "text" "integer" "integer" "integer" ...
#> ..$ Categorical variable : chr [1:21] "yes" "yes" "yes" "yes" ...
#> ..$ Categories in data dictionary : chr [1:21] "[1] Study 1\n[2] Study 2\n[3] Study 3\n[4] Study 4\n[5] Study 5" "[0] Single (never married) or not living with partner\n[1] Married or living with partner\n[2] Divorced or sepa"| __truncated__ "[0] Single (never married) or not living with partner\n[1] Married or living with partner\n[2] Divorced or sepa"| __truncated__ "[0] Single (never married) or not living with partner\n[1] Married or living with partner\n[2] Divorced or sepa"| __truncated__ ...
#> ..$ Non-valid categories : chr [1:21] NA NA NA NA ...
#> ..$ Number of rows : chr [1:21] "13500" "4000" "3000" "1000" ...
#> ..$ Number of valid values : chr [1:21] "13500" "3994" "2961" "0" ...
#> ..$ Number of non-valid values : chr [1:21] "0" "0" "0" "0" ...
#> ..$ Number of empty values : chr [1:21] "0" "6" "39" "1000" ...
#> ..$ % Valid values : chr [1:21] "100" "99.85" "98.7" "0" ...
#> ..$ % Non-valid values : chr [1:21] "0" "0" "0" "0" ...
#> ..$ % Empty values : chr [1:21] "0" "0.15" "1.3" "100" ...
#> ..$ Number of distinct values : chr [1:21] "5" "3" "3" "0" ...
#> ..$ Values present in dataset : chr [1:21] "Valid values : \n[1] Study 1 : 29.63%\n[2] Study 2 : 22.22%\n[3] Study 3 : 7.41%\n[4] Study 4 : 18.52%\n[5] Study 5 : 22.22%\n" "Valid values : \n[0] Single (never married) or not living with partner : 4.12%\n[1] Married or living with part"| __truncated__ "Valid values : \n[0] Single (never married) or not living with partner : 2.4%\n[1] Married or living with partn"| __truncated__ "Valid values : \n[0] Single (never married) or not living with partner : 0%\n[1] Married or living with partner"| __truncated__ ...
#> ..$ Data dictionary categories not present in dataset: chr [1:21] NA "[3] Widowed\n" "[3] Widowed\n" "[0] Single (never married) or not living with partner\n[1] Married or living with partner\n[2] Divorced or sepa"| __truncated__ ...
#> ..$ Dataset values not present in data dictionary : chr [1:21] NA NA NA NA ...
#> $ Harmo dataset assessment : tibble [1,155 × 4] (S3: tbl_df/tbl/data.frame)
#> ..$ Index : chr [1:1155] "1" "1" "1" "1" ...
#> ..$ Variable name : chr [1:1155] "adm_unique_id" "adm_unique_id" "adm_unique_id" "adm_unique_id" ...
#> ..$ Harmo dataset assessment: chr [1:1155] "[INFO] - Duplicated rows (excluding identifier)." "[INFO] - Duplicated rows (excluding identifier)." "[INFO] - Duplicated rows (excluding identifier)." "[INFO] - Duplicated rows (excluding identifier)." ...
#> ..$ Value : chr [1:1155] "PID2444 ; PID6070" "PID5531 ; PID7645" "PID2866 ; PID1861 ; PID8315" "PID3435 ; PID3700" ...
#> - attr(*, "madshapR_group::group_by")= chr "adm_study_id"