---
title: "ReadMe - V_air_temperature_meadow"
author: "| Eeva Soininen \n| (eeva.soininen@uit.no)\n"
date: "`r format(Sys.time(), '%d.%m.%Y')`"
output:
  pdf_document:
    highlight: tango
    includes:
      in_header: highlight_echo.tex
    number_sections: true
  html_document:
    self_contained: no
  word_document: default
header-includes:
- \usepackage{setspace}\singlespacing
- \usepackage{enumitem}
- \usepackage{float}
- \floatplacement{table}{H}
- \usepackage[skip=2pt, labelfont=bf]{caption}
- \usepackage{titling}
- \setlength{\droptitle}{-6em}
- \usepackage{xcolor}
- \definecolor{bgcolor}{HTML}{F8F8F8}
- \let\oldtexttt\texttt
- \renewcommand{\texttt}[1]{ \colorbox{bgcolor}{\oldtexttt{#1}} }
- \usepackage{framed}
- \definecolor{shadecolor}{HTML}{F8F8F8}
- \pretitle{\begin{center}\LARGE}
- \posttitle{\par\end{center}\vskip 0.1em}
- \usepackage{colorprofiles}
- \usepackage[a-1b]{pdfx}
urlcolor: blue
---

```{r setup, echo=FALSE, warning=FALSE, message=FALSE}

#### set up

## load libraries
library(knitr)
library(kableExtra)
library(xlsx)


## set directories
in.dir<-"C:/Users/hbo042/Box/COAT/Data Management/Formatted data/Climate module/V_air_temperature_meadow/data/2019" # path to folder with temperature data files
meta.dir<-"C:/Users/hbo042/Box/COAT/Data Management/Formatted data/Climate module/V_air_temperature_meadow"  # path to folder with metadta

## choose dataset
dataset<-"V_air_temperature_meadow"

## load data
filenames<-dir(paste(in.dir, sep = "/"))

mylist<-c()
for (i in 1:length(filenames)){
  mylist[[i]]<-read.table(paste(in.dir, filenames[i], sep = "/"), header=T, sep = ";")
}

myfile<-do.call(rbind, mylist)  # combine all files

## load metadata
meta_list<-c()
meta_names<-dir(paste(meta.dir, "metadata", sep = "/"))

for (i in 1:length(meta_names)) {
  meta_list[[i]]<-read.delim(paste(meta.dir, "metadata", meta_names[i], sep = "/"), header = T, sep = ";")
}

meta<-do.call(rbind, meta_list)

## load coorinate and aux files
coord<-read.table(paste(meta.dir, paste(dataset, "_coordinates.txt", sep = ""), sep = "/"), header = TRUE, sep = ";")
aux<-read.table(paste(meta.dir, paste(dataset, "_aux.txt", sep = ""), sep = "/"), header = TRUE, sep = ";")

```

# Protocol

Air temperature is measured using iButtons (Maxim Integrated, San Jose, California, USA) with a loggin interval of 4 hours following the COAT protocol 'protocol_temperature_loggers_air_temperature_varanger'. 

## Timing of sampling and changes in that
Temperature loggers are usually deployed the whole year and are downloaded once a year. However, there is some missing data in summer (between a few days and ca 2 month) when the loggers were changed.

## Spatial layout of sampling and changes in that
From 2005 onwards, temperature loggers were deployed in two localities, Vestre Jakobselv and Komagdalen.
In Komagdalen, a new section ('sandfjorddalen') has been included since 2010.

A third locality, Ifjordfjellet, has been included from 2009 to 2017.  

From 2005 to 2011, temperature loggers were deployed in the first and the last site of each section. From 2011 onwards, temperature loggers were deployed in all sites.

The loggers were deployed in 3 different setups:  \newline
\textbf{2005-2010:} Two loggers in 100 cm height at the thicket edge of each site. These loggers are named siteID_th1_100 and siteID_th2_100. \newline
\textbf{2010-2011:} Two loggers at the thicket edge of each site. One logger was deployed at ground level and one in 30 cm height. These loggers are named siteID_th (ground level) and site_ID_th30 (30 cm height). \newline
\textbf{2011 onwards:} Two loggers at ground level in each site. One logger is deployed at the thicket edge and one at the meadow edge (see protocol). These loggers are named siteID_th (thicket edge) and siteID_me (meadow edge).

If loggers were moved during winter and were not collected at the correct position, they are named siteID_na. If both loggers were found at the wrong position, they are named siteID_na1 and siteID_na2.

Note: the names thicket and meadow edge refer to the postion of the logger within the sampling quadrat, sites in the section 'sandfjorddalen', iesjohka' and 'giksjohka' do not have thicket habitats.

More detailed information about which sites were included in the study design and the different setups can be found in the auxialiary file '`r paste(dataset, "aux.txt", sep = "_")`'.\newline


# Description of the dataset
The dataset includes four differnt types of files and all files are saved as ;-separated txt-files:
\begin{itemize}[topsep=0pt, itemsep=0.5pt]
\item One data file per year (\verb|loggerID_YEAR.txt|)
\item One meatadata file per year with dates when the loggers were deployed and collected \newline
(\verb|V_air_temperature_meadow_metadata_YEAR.txt|)
\item One coordinate file with coordinates of all sites (\verb|V_air_temperature_meadow_coordinates.txt|)
\item One auxiliary file with information about which sites are included in the study design \newline
(\verb|V_air_temperature_meadow_aux.txt|)
\end{itemize}

## Temperature data files
These files contain temperature values measured every four hours.

\vspace{5mm}
**Example of the first rows of the data files:**

```{r, echo=FALSE, include=TRUE, results="markup", comment=NA}
### First 3 rows of a data file
head(myfile, n=3)
```


**Description of the columns included in the data files:**
```{r, echo=FALSE, include=TRUE}
### Make a description table

des<-data.frame(`Column name`=colnames(myfile), 
                `Description`=c("Study region",
                                "Locality (within region)",
                                "Section (within locality)",
                                "Habitat type", 
                                "Unique Site ID",
                                "Corner of the sampling quadrat in which the logger was placed", 
                                "Logging date",
                                "Logging time",
                                "Consistent logging date and time for all loggers",
                                "Logger ID",
                                "Height in which the logger was deployed (cm)", 
                                "measured temperature (°C)",
                                "Comments"),
                `Possible values`=c("varanger",
                                    paste(unique(myfile$sn_locality), collapse = ", "),
                                    paste(unique(meta$sn_section), collapse = ", "),
                                    paste(unique(myfile$sc_type_of_sites_ecological), collapse = ", "),
                                    paste("e.g.", paste(unique(myfile$sn_site)[c(1, 5, 10, 20, 24)], collapse = ", ")),
                                    paste(unique(myfile$sn_plot), collapse = ", "),
                                    "yyyy-mm-dd",
                                    "hh:mm:ss",
                                    "yyyy-mm-dd hh:mm:ss",
                                    paste("e.g.", paste(unique(myfile$v_logger_id)[c(1, 5, 10, 20, 24)], collapse = ", ")),
                                    "0, 30, 100",
                                    "[numeric]",
                                    "[character]"))

## edit column names
colnames(des)<-c("Column name", "Description", "Possible values")

## include description table in output file
kable(des,
      longtable=TRUE, format = "latex", booktabs="TRUE", linesep = "\\addlinespace")  %>% 
  column_spec(c(2,3), c("5cm"))
```


## Metadata files
These files contain additional inforamtion for each logger, such as height of the logger and dates when the loggers were deployed and collected.

\vspace{5mm}
**Example of the first rows of metadata files:**

```{r, echo=FALSE, include=TRUE, results="markup", comment=NA}
### First 3 rows of the data files:
head(meta, n=3)
```


**Description of the columns included in the metadata files:**
```{r, echo=FALSE, include=TRUE}
### Make a description table

des<-data.frame(`Column name`=colnames(meta), 
                `Description`=c("Study region",
                                "Locality (within region)",
                                "Section (within locality)",
                                "Habitat type", 
                                "Unique Site ID",
                                "Corner of the sampling quadrat in which the logger was placed",
                                "Year in which loggers were changed",
                                "Logger ID",
                                "Height in which the logger was deployed (cm)", 
                                "Date when the logger was collected",
                                "Date when the logger was deployed",
                                "Date when the logger was started",
                                "Initials of observer",
                                "Comments"),
                `Possible values`=c("varanger",
                                    paste(unique(meta$sn_locality), collapse = ", "),
                                    paste(unique(meta$sn_section), collapse = ", "),
                                    paste(unique(meta$sc_type_of_sites_ecological), collapse = ", "),
                                    paste("e.g.", paste(unique(meta$sn_site)[c(1, 5, 10, 20, 24)], collapse = ", ")),
                                    paste(unique(meta$sn_plot), collapse = ", "),
                                    "e.g. 2005",
                                    paste("e.g.", paste(unique(meta$v_logger_id)[c(1, 5, 10, 20, 24)], collapse = ", ")),
                                    "0, 30, 100",
                                    "yyyy-mm-dd",
                                    "yyyy-mm-dd",
                                    "yyyy-mm-dd",
                                    "e.g. es (Eeva Soininen)",
                                    "[character]"))

## edit column names
colnames(des)<-c("Column name", "Description", "Possible values")

## include description table in output file
kable(des,
      longtable=TRUE, format = "latex", booktabs="TRUE", linesep = "\\addlinespace")  %>% 
  column_spec(c(2,3), "5cm")
```



## Coordinate file
This file contains the coordinates of all sites included in the study desgin. Coordinates are given in decimal degrees and UTM 33 (WGS 84).

\vspace{5mm}
**Example of the first rows of coordinate files:**
```{r, echo=FALSE, include=TRUE, results="markup", comment=NA}
### First 3 rows of the plant abundance dataset
head(coord, n=3)
```

## Auxiliary file
This file contains further information about the dataset such as old site names (for example used in raw data files before 2019) and the years when sites were first included in the study design and when sites were excluded from the study design. It also includes in which sites were included in which of the different setups (see section 1.2).

\vspace{5mm}
**Example of the first rows of auxiliary-files:**
```{r, echo=FALSE, include=TRUE, results="markup", comment=NA}
### First 3 rows of the plant abundance dataset
head(aux, n=3)
```


# Data cleaning and formatting
From 2005 to 2008, rawdata has been cleaned by Vegard Trasti. From 2009 to 2018, rawdata has been cleaned and days when the loggers were not not deployed were removed by Hanna Boehner. All pre-cleaned files were formatted meeting the requirements of the COAT dataportal by Hanna Boehner.\newline

From 2019 onwards, raw metdata (entered in an excel-template) and temperature data (downloaded from the temperature loggers) are cleaned and formatted using the scripts \newline `01_clean_metadata_temperature_data_meadows_COAT.R` and \newline
`02_clean_cut_temperature_data_meadows_COAT.R`. \newline

\textbf{1. Metadata:} The scripts checks all variables for correct spelling and correct formatting. Errors will be corrected in the script and the data will be reformatted according to the requirements of the COAT data portal. The file will be saved as a ;-spearated txt-file and uploaded to to COAT data portal. In particular, the script checks for:

\begin{itemize}[topsep=0pt, itemsep=0.5pt]
\item \textbf{sn\_site:} Correct spelling of all variables.
\item \textbf{t\_date:} Correct format (yyyy-mm-dd), dates in other formats are reformatted.
\item \textbf{v\_logger\_id:} Missing observations. Are all loggers included in the metadata?
\item \textbf{v\_observer:} Correct format (initials and lowercase letters), observer is reformatted e.g. if full names were used.
\item \textbf{v\_comment:} Correct spelling and format (lowercase letters and english). Comments are edited or translated if necessary.

\end{itemize}

All corrections that lead to differences between rawdata and cleaned data are double-checked in the fieldbooks and a comment is added. \newline

\textbf{2. Temperature data:} The script removes the days when the logger was not deployed in the field , reformats the data according to the requirements of the COAT data portal and saves the data as ;-separated txt-tiles which can be uploaded to the COAT data portal. 
In particular, the script  includes the following  steps:

\begin{itemize}[topsep=0pt, itemsep=0.5pt]
\item Checking if the logger was started before it was deployed in the field and was stopped after it was collected. 
\item Removing days when the logger was not deployed (based on t\_date\_logger\_in and t\_date\_logger\_out in the metadata).
\item Reformatting the columns t\_date and t\_time and adding t\_bintime.
\item Checking the recorded temperature values. Sometimes the logger jumps to extreme values (above +40 °C and below - 40 °c), these will be set to NA.
\item Adding all other necessary columns such as spatial variables and loggerID.
\end{itemize}


