methods.rmd

---
title: "Methods"
output:
  html_document:
    fig_cap: yes
    highlight: tango
    smooth_scroll: no
    theme: flatly
    toc: yes
    toc_float: yes
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

The idea of the [Typological Atlas of Daghestan](https://lingconlab.ru/tald) was to create a tool for the visualization of linguistic features typical of the languages of Daghestan.

Although the language sample underlying [TALD](https://lingconlab.ru/tald) has grown to include a number of neighboring languages (more on that in [Language sample](#Language-sample) below), the core of the project remains Daghestan.

Our aim is to achieve maximum coverage of languages and dialects. Datasets on particular topics should be updatable, so that our map visualizations become incrementally more accurate. In order to make the resource updatable, contributors can choose to maintain the right to approve or reject any proposed updates or corrections, or they can choose to waive that right. Note that if you want to stay involved, this also entails a responsibility to review and reply to propositions within a certain time frame. See [Step 5](steps.html) on how to propose changes to a dataset or chapter that is already published. 

Data on linguistic features is collected primarily from descriptive literature. As a result, the Atlas can also be helpful in bibliographical research.

# Datapoints

The initial approach of [TALD](https://lingconlab.ru/tald) assigned one value for a linguistic feature to each language, based on a representative doculect. 

This information could then be mapped onto generalized language datapoints, or all villages where the language in question is spoken.

Below are two possible visualizations for the same dummy feature: the initial consonant of various cognates meaning 'bridge'.

```{r, echo=FALSE,out.width="40%", out.height="20%",fig.cap="General language datapoints vs. village datapoints",fig.show='hold',fig.align='center'}

knitr::include_graphics(c("images/walsex.png","images/taldex.png"))
``` 

<center>
|language|feature                      |value|form |
|--------|-----------------------------|-----|-----|
|Avar    |Initial consonant of 'bridge'|ƛ'   |ƛ'o  |
|Khwarshi|Initial consonant of 'bridge'|t'   |t'eru|
|Karata  |Initial consonant of 'bridge'|ƛ'   |ƛ'eru|
</center>

A benefit of the visualization on the right is that it shows the distribution and size of language communities more accurately. A drawback is that it leads to gross overgeneralization and erases dialectal differences, because it is not based on more data than the visualization on the left. 

All of the Avar villages, for example, are colored according to data from Standard Avar, while we know that 'bridge' in the Zaqatala dialect spoken in Northern Azerbaijan is pronounced *kːjo*. These villages should thus have a different value.

# Current approach

To improve the accuracy of our visualizations, we currently collect all attested values for a given feature, taking into account any idiom we have data on, including standard languages, dialects spoken in multiple villages and single-village idioms. 

**The visualization shows the most accurate level of granularity available for each village / point on the map.**

For example, the Andi language is spoken in 17 villages. There are 9 main villages, each of which has its own idiom. These can be divided into two main dialects: Upper Andi and Lower Andi. In addition, there are 8 villages for which we have no information on the variety spoken there.

Now let us look at a relatively straightforward linguistic feature like **Number of noun classes**, for which we have general data on both the Upper and Lower dialects, and more accurate information on several villages from the Upper group. One of these villages (Rikvani) even has a value that differs from the other varieties we have data for.

The table below summarizes the different values observed for the language, divided by type of idiom (the number of noun classes is indicated between brackets and each value is color-coded).

|Language|Toplevel dialect|Village|
|--------|----------------|-------|
|Andi <span style="color:Firebrick">●</span> (5)  |Upper Andi <span style="color:Firebrick">●</span>  (5)    |Rikvani <span style="color:Thistle">●</span> (6)|
|        |Lower Andi <span style="color:MistyRose">●</span> (3)    |       |

The diagram below shows the dialect grouping of Andi villages and their values for the noun class feature. At the center is the language as a whole: it has the same value as the Upper group and the eponymous village dialect of Andi, which are most representative of the language as a whole. On the map, the unclassified Andi villages (colored grey in the scheme below) will be colored according to the general language information.

```{r, echo=FALSE}

library(DiagrammeR)

DiagrammeR::grViz("digraph {
  
graph[layout = neato, rankdir = LR]

node [fontsize = 14,
      shape = oval, 
      style = filled, 
      fillcolor = Firebrick, 
      fontcolor = white,
      color = Firebrick]
        
# language      
      Andi

node [fontsize = 10]

# dialects
      Upper
      
node [shape = oval, 
      style = filled, 
      fillcolor = WhiteSmoke, 
      fontcolor = black,
      color = WhiteSmoke]
      Other
      
node [shape = oval, 
      style = filled, 
      fillcolor = MistyRose, 
      fontcolor = black,
      color = MistyRose]
      
      Lower

# Andi-Upper-Villages

node [fontsize = 8,
      shape = oval,
      style = filled, 
      fillcolor = Firebrick, 
      fontcolor = white,
      color = Firebrick]
Chanko
Zilo
Ashali
Andiv [label = 'Andi']
Gunkha
Gagatli

# Rikvani

node [shape = oval,
      style = filled, 
      fillcolor = Thistle, 
      fontcolor = black,
      color = Thistle]
Rikvani

# Lower villages

node [shape = oval, 
      style = filled, 
      fillcolor = MistyRose, 
      fontcolor = black,
      color = MistyRose]

Kvankhidatli
Muni

# Other villages

node [shape = oval, 
      style = filled, 
      fillcolor = WhiteSmoke, 
      fontcolor = black,
      color = WhiteSmoke]
Mekheturi
Shivor
Khando
Rushukha
Novogagatli
Aytkhan
Dzhugut
Tsibilta

edge [color = black, arrowhead = none]

Andi -> {Other, Upper, Lower}

Other -> {Mekheturi, Shivor, Rushukha, Khando, Tsibilta, Novogagatli, Aytkhan, Dzhugut}
Upper -> {Andiv, Rikvani, Gagatli, Zilo, Ashali, Chanko, Gunkha} 
Lower -> {Muni, Kvankhidatli}
    
}")


```

The coverage of the Atlas is far from sufficient because we lack data for many dialects, and we do not know the dialect affiliation of a large number of villages in the area. We compensate for this shortcoming by encoding the level of accuracy/granularity for each datapoint, and allowing the user to toggle which levels to display (see [Map visualization](#Map-visualization) below). Ideally, our datasets will be updated when new information becomes available.

# Map visualization{.tabset .tabset-fade .tabset-pills}

[TALD](https://lingconlab.ru/tald) currently offers three different map visualizations:

1. **General datapoints** displays one datapoint for each language in the sample, showing the language affiliation (inner dot) and the value for the linguistic feature (outer dot) for each point.
2. **Extrapolated data** represents each language as a cluster of dots, which correspond to villages where a certain language is spoken.
3. **Data granularity** allows the user to show only certain levels of data accuracy for the feature. For example, you can uncheck "language" to remove all the dots that were colored according to general information about the language in the absence of more accurate data. In this case it will remove all Andi villages that have no dialect classification.

You can click on a datapoint to view a pop-up window with the name of the language (with a link to the [Glottolog](https://glottolog.org) database), the village, the granularity of data used to color this datapoint, and the value.

```{r, echo=FALSE, message=FALSE}

# packages

library(tidyverse)
library(lingtypology)

# load data

feature <- read_tsv("dummy_data/dummy_feature.csv")
villages <- read_tsv("dummy_data/dummy_villages.csv")

# remove data not for mapping

feature <- feature[(feature$map == "yes"),]

# split feature data into dialect levels

feature_group <- feature %>%
  group_by(type) %>%
  group_split()

feature_tl <- data.frame(feature_group[[1]])
feature_v <- data.frame(feature_group[[2]])

feature_tl$granularity <- "toplevel dialect"
feature_v$granularity <- "village dialect"

# merge feature data with villages dataset

## create matching columns

colnames(feature_tl)[colnames(feature_tl) == "idiom"] <- "toplevel_dialect"
colnames(feature_v)[colnames(feature_v) == "idiom"] <- "village_dialect"

## toplevel dialect data

tlevel_villages <- merge(villages, feature_tl, by = "toplevel_dialect")
v_villages <- merge(villages, feature_v, by = "village_dialect")

# [this generates a lot of garbage columns]

## combine dialect data of different granularity

dialect_villages <- full_join(v_villages, tlevel_villages, by = "village")

# [not very convenient, data from different datasets is in different columns]

## do a simple rbind instead

dialect_villages2 <- rbind(v_villages, tlevel_villages)

dialect_villages3 <- dialect_villages2[!duplicated(dialect_villages2$village),]

# [this gives the desired result, but it's cumbersome and will fail 
# if we happen to have multiple villages with the same name in our set]

# IMPORTANT: SETS SHOULD BE MERGED IN THE RIGHT ORDER (HIGH GRAN - LOW GRAN)
# SO THAT THE DUPLICATES WITH THE HIGHEST GRANULARITY ARE KEPT

## and now for the villages for which we have no data

# [*adds column for villages that lack a dialect affiliation*]

### isolate general language data

feature_l <- feature %>%
  filter(genlang_point == "yes") %>%
  mutate(granularity = "language") %>%
  mutate(default_level = lang) %>%
  select(-idiom)

### create a set of unaffiliated villages

#lost_villages <- villages[villages$default_level == "yes",]

### make them match

### merge feature data and village set

lang_villages <- merge(villages, feature_l, by = "default_level")

### add to the refined set

alldata <- full_join(dialect_villages3, lang_villages, by = "village")

### и еще раз волшебный rbind

alldata2 <- rbind(dialect_villages3, lang_villages)
alldata3 <- alldata2[!duplicated(alldata2$village),]

```

## 1. General datapoints

```{r, echo=FALSE, message=FALSE}

map.feature(feature_l$lang,
            features = feature_l$lang,
            color = "#003366",
            title = "Language",
            label = feature_l$lang,
            stroke.features = as.factor(feature_l$value),
            stroke.color = c("MistyRose", "Firebrick", "Thistle"),
            stroke.title = feature_l$feature[1],
            zoom.control = T,
            zoom.level = 7)

```

## 2. Extrapolated data

```{r, echo=FALSE, message=FALSE}

map.feature(alldata3$lang.x,
            latitude = alldata3$lat,
            longitude = alldata3$lon,
            features = alldata3$lang.x,
            color = "#003366",
            title = "Language",
            label = alldata3$village,
            stroke.features = as.factor(alldata3$value),
            stroke.color = c("MistyRose", "Firebrick", "Thistle"),
            stroke.title = alldata3$feature[1],
            popup = paste("<b>Village:</b>", alldata3$village, "<br>",
                          "<b>Data:</b>", alldata3$granularity, "<br>",
                          "<b>Value:</b>", alldata3$value),
            zoom.control = T)

```

## 3. Data granularity

```{r, echo=FALSE, message=FALSE}

# единсвтенное, мне хотелось бы, чтобы можно было посмотреть в popup еще и название идиома рядом с его левелом, например: Upper, toplevel dialect, но не знаю как это реализовать

map.feature(alldata3$lang.x,
            latitude = alldata3$lat,
            longitude = alldata3$lon,
            features = as.factor(alldata3$value),
            color = c("MistyRose", "Firebrick", "Thistle"),
            width = 10,
            title = alldata3$feature[1],
            legend.position = "bottomleft",
            label = alldata3$village,
            control = alldata3$granularity,
            popup = paste("<b>Village:</b>", alldata3$village, "<br>",
                          "<b>Data:</b>", alldata3$granularity, "<br>",
                          "<b>Value:</b>", alldata3$value),
            zoom.control = T)

```

# Language sample

As mentioned earlier, [TALD](https://lingconlab.ru/tald) was originally conceived of as a resource about the languages of Daghestan. 

[The East Caucasian villages dataset](https://raw.githubusercontent.com/LingConLab/TALD/refs/heads/master/data/tald_villages.csv) -- a dataset that contains a list of villages in the eastern Caucasus, their coordinates, and the languages spoken there -- was created as a basis for visualizations in the Atlas. Initially it covered all villages of Daghestan and some East Caucasian speaking communities in Georgia and Northern Azerbaijan.

Villages of Chechnya and Ingushetia, and several more communities in Azerbaijan and Georgia, were added later. It was relevant to include these extra datapoints outside of Daghestan for the development of areal hypotheses.

[Here](https://lingconlab.ru/tald/languages.html) is a list of the languages included in our sample.