---
title: "Raw Data Import"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Raw Data Import}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

## Introduction

The aim of this document is to outline the basic workflow of importing data downloaded from the [ICES Regional Database & Estimation System (RDBES)](https://rdbes.ices.dk/#/) or a `list` object containing data frames (or data.tables) into `R` using the `RDBEScore` package.

The function **createRDBESDataObject** is intended to directly import Commercial Landing (CL), Commercial Effort (CE) and Commercial Sampling (CS) tables downloaded from [RDBES](https://rdbes.ices.dk/#/).

## Load the package

```{r setup}
library(RDBEScore)
```


## Importing zipped files

It can directly import the `.zip` archive from the RDBES download containing all mandatory hierarchy tables plus VD and SL:

```{r}
importedH1 <- createRDBESDataObject(input = "./vignetteData/H1_2023_10_16.zip")
#print the not NULL table names
names(importedH1[!unlist(lapply(importedH1, is.null))])
```
The easiest way to get a glimpse of the imported data hierarchy and single table row counts is just to print it. The information also includes the range of number sampled and number total for each table together with the selection method and number of rows.
```{r}
#calls the print function 
importedH1
```


It can import the CL, CE, VD or SL tables `.zip` archives, but will include all other tables as `NULL`:

```{r}
importedSL <- createRDBESDataObject(input = "./vignetteData/HSL_2023_10_16.zip")
#print the not NULL table names
importedSL
```

It can also handle overwriting `zip` file original files with files appearing later in the list. However each overwrite results in a warning!

```{r}
importFiles <- c("./vignetteData/HSL_2023_10_16.zip",
                 "./vignetteData/H1_2023_10_16.zip")
importedTables <- createRDBESDataObject(input = importFiles)
#print the not NULL table names
names(importedTables[!unlist(lapply(importedTables, is.null))])
```

## Importing csv files

It can also import the unzipped `.csv` files with the default RDBES names:

```{r}
importedVS <- createRDBESDataObject(input = "./vignetteData/",
                                    listOfFileNames = list("VS" = "VesselSelection.csv"))
#print the not NULL table names
names(importedTables[!unlist(lapply(importedTables, is.null))])
```

## Importing list of data frames

It can also import a `list` object containing data frames (or data.tables). However, it should be noted that this type of import bypasses the RDBES upload data integrity checks.

```{r}
#list of data frames 
listOfDfsH1 <- readRDS("./vignetteData/H1_2023_10_19.rds")
#print the class of the list elements 
sapply(listOfDfsH1, class)
```

```{r}
importedList <- createRDBESDataObject(listOfDfsH1)
```

## Object class RDBESDataObject

It should be noted that the objects created are of the S3 class "**RDBESDataObject**". The class has defined **print()**, **summary()** and **sort()** methods. For more info on theese see vignette [Manipulating RDBESDataObjects](manipulating-rdbesdataobjects.html). 

```{r}
importedTables <- createRDBESDataObject("./vignetteData/H1_2023_10_16.zip")
class(importedTables)
```


### validate RDBESDataObject

RDBESDataObject structure can be validated using the **validateRDBESDataObject()** function.

```{r}
validateRDBESDataObject(importedTables, verbose = TRUE)
```


To see what you can do with the imported **RDBESDataObject** see other vignettes like [Manipulating RDBESDataObjects](manipulating-rdbesdataobjects.html).

Other vignettes:

```{r echo=FALSE, results='asis'}
vignettes <- as.data.frame(browseVignettes(package = "RDBEScore")$RDBEScore)
for (i in seq_along(vignettes$PDF)) {
  cat(sprintf("- [%s](%s)\n", vignettes$Title[i], vignettes$PDF[i]))
}

#END
```