R/02-harmo_process_harmonization.R
harmo_process.Rd
Reads a DataSchema and Data Processing Elements to generate a
harmonized dossier from input dataset(s) in a dossier and associated
metadata. The function
has one argument that can optionally be declared by the user
(unique_col_dataset
). It refers to the columns which contains name of
each harmonized dataset. These two columns are added to ensure that there
is always a unique entity identifier when datasets are pooled.
harmo_process(
object = NULL,
dataschema = attributes(dossier)$`Rmonize::DataSchema`,
data_proc_elem = attributes(dossier)$`Rmonize::Data Processing Elements`,
harmonized_col_dataset = attributes(dossier)$`Rmonize::harmonized_col_dataset`,
harmonized_col_id = attributes(dossier)$`Rmonize::harmonized_col_id`,
.debug = FALSE,
dossier = object
)
Data frame(s) or list of data frame(s) containing input dataset(s).
A DataSchema object.
A Data Processing Elements object.
A character string identifying the column to use for dataset names.
A character string identifying the name of the column present in every dataset to use as a participant identifier.
Allow user to test the inputs before processing harmonization.
A list of data frame(s) containing harmonized dataset(s). The DataSchema and Data Processing Elements are preserved as attributes of the harmonized dossier.
A dossier is a named list containing one or more data frames, which are
input datasets. The name of each data frame in the dossier will be used as
the name of the associated harmonized dataset produced by harmo_process()
.
A DataSchema is the list of core variables to generate across datasets and
related metadata. A DataSchema object is a list of data frames with elements
named 'Variables' (required) and 'Categories' (if any). The 'Variables'
element must contain at least the name
column, and the 'Categories'
element must contain at least the variable
and name
columns to be usable
in any function. In 'Variables' the name
column must also have unique
entries, and in 'Categories' the combination of variable
and name
columns
must also be unique.
The Data Processing Elements specifies the input elements and processing algorithms
to generate harmonized variables in the DataSchema formats. It is also
contains metadata used to generate documentation of the processing.
A Data Processing Elements object is a data frame with specific columns
used in data processing: dataschema_variable
, input_dataset
,
input_variables
, Mlstr_harmo::rule_category
and Mlstr_harmo::algorithm
.
To initiate processing, the first entry must be the creation of a harmonized
primary identifier variable (e.g., participant unique ID).
{
# Use Rmonize_examples to run examples.
library(dplyr)
library(stringr)
library(lubridate)
# Perform data processing
dossier <- Rmonize_examples[str_detect(names(Rmonize_examples),"input_dataset_study")]
names(dossier) <- str_remove(names(dossier), "input_")
dataschema <- Rmonize_examples$`DataSchema`
data_proc_elem <- Rmonize_examples$`Data_Processing_Element_no errors`
harmonized_dossier <- harmo_process(
dossier,
dataschema,
data_proc_elem,
harmonized_col_dataset = 'adm_study_id')
glimpse(harmonized_dossier$dataset_study1)
}
#>
#> Attaching package: 'lubridate'
#> The following objects are masked from 'package:base':
#>
#> date, intersect, setdiff, union
#> - Data Processing Elements: ------------------------------------------------
#>
#> --harmonization of : dataset_study1 -----------------------------------
#> processing 1/9 : adm_unique_id id created
#> processing 2/9 : adm_study_id complete
#> processing 3/9 : sdc_age_m complete
#> processing 4/9 : sdc_marital_m complete
#> processing 5/9 : lsb_alc_binge_m_preg complete
#> processing 6/9 : lsb_alc_binge_m_first_tri impossible
#> processing 7/9 : preg_gestational_age_del complete
#> processing 8/9 : sdc_sex_c complete
#> processing 9/9 : pm_birthweight complete
#>
#> --harmonization of : dataset_study2 -----------------------------------
#> processing 1/9 : adm_unique_id id created
#> processing 2/9 : adm_study_id complete
#> processing 3/9 : sdc_age_m complete
#> processing 4/9 : sdc_marital_m complete
#> processing 5/9 : lsb_alc_binge_m_preg impossible
#> processing 6/9 : lsb_alc_binge_m_first_tri impossible
#> processing 7/9 : preg_gestational_age_del complete
#> processing 8/9 : sdc_sex_c complete
#> processing 9/9 : pm_birthweight complete
#>
#> --harmonization of : dataset_study3 -----------------------------------
#> processing 1/9 : adm_unique_id id created
#> processing 2/9 : adm_study_id complete
#> processing 3/9 : sdc_age_m complete
#> processing 4/9 : sdc_marital_m impossible
#> processing 5/9 : lsb_alc_binge_m_preg complete
#> processing 6/9 : lsb_alc_binge_m_first_tri impossible
#> processing 7/9 : preg_gestational_age_del complete
#> processing 8/9 : sdc_sex_c complete
#> processing 9/9 : pm_birthweight complete
#>
#> --harmonization of : dataset_study4 -----------------------------------
#> processing 1/9 : adm_unique_id id created
#> processing 2/9 : adm_study_id complete
#> processing 3/9 : sdc_age_m complete
#> processing 4/9 : sdc_marital_m complete
#> processing 5/9 : lsb_alc_binge_m_preg complete
#> processing 6/9 : lsb_alc_binge_m_first_tri complete
#> processing 7/9 : preg_gestational_age_del complete
#> processing 8/9 : sdc_sex_c complete
#> processing 9/9 : pm_birthweight complete
#>
#> --harmonization of : dataset_study5 -----------------------------------
#> processing 1/9 : adm_unique_id id created
#> processing 2/9 : adm_study_id complete
#> processing 3/9 : sdc_age_m complete
#> processing 4/9 : sdc_marital_m complete
#> processing 5/9 : lsb_alc_binge_m_preg **ERROR**
#> processing 6/9 : lsb_alc_binge_m_first_tri complete
#> processing 7/9 : preg_gestational_age_del complete
#> processing 8/9 : sdc_sex_c complete
#> processing 9/9 : pm_birthweight complete
#>
#>
#> - CREATION OF HARMONIZED DATA DICTIONARY : --------------------------------
#>
#> dataset_study1 : done
#> dataset_study2 : done
#> dataset_study3 : done
#> dataset_study4 : done
#> dataset_study5 : done
#>
#>
#> ------------------------------------------------------------------------------
#>
#> The Data Processing Elements contain 'error' statuses. Dataset with any 'error'
#> status will appear as empty in the harmonized dossier. When the harmonization
#> statuses are finalized, remember to rerun the process with the updated
#> Data Processing Elements and generate the final harmonized datasets.
#>
#>
#> Useful tip: If you identified errors and want to correct them later, you can specify
#> 'undetermined' in the column 'Mlstr_harmo::algorithm' of your problematic
#> Data Processing Elements(s). Such Data Processing Elements will be ignored.
#>
#> - WARNING MESSAGES (if any): ----------------------------------------------
#>
#> Rows: 4,000
#> Columns: 9
#> $ adm_unique_id <chr> "PID7216", "PID5380", "PID5531", "PID7612", …
#> $ adm_study_id <chr+lbl> "1", "1", "1", "1", "1", "1", "1", "1", …
#> $ sdc_age_m <int> 27, 38, 27, 26, 33, 31, 32, 29, 29, 26, 33, …
#> $ sdc_marital_m <int+lbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
#> $ lsb_alc_binge_m_preg <int+lbl> 0, 0, 0, 0, NA, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ lsb_alc_binge_m_first_tri <int+lbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
#> $ preg_gestational_age_del <int> 41, 38, 39, 41, 40, 38, 38, 39, 41, 40, 41, …
#> $ sdc_sex_c <int+lbl> 1, 2, 1, 1, 1, 2, 1, 1, 1, 1, 1, 2, 1, 1…
#> $ pm_birthweight <dbl> 2603.164, 3295.346, 3426.888, 3112.095, 2394…