---
title: "Word Clouds in R"
author: "Sean Carver"
date: "November 12, 2016"
output: html_document
---

## Last Few Weeks of the Semester

Folks, we are essentially finished with the topics that I usually cover in Basic Statistics.

Since the last exam we have covered:

1. Sampling distributions for counts and proportions.
2. Confidence intervals.

What else do I sometimes cover?

1. Computations with standard deviations of random variables (the rest of homework 17).
2. T-tests (homework 22).
3. Use and Abuse of Tests of Significance (homework 22)
4. Power and Sample-Size (homework 23).

What else could we do that is basic statistics?

1. Analysis of Variance (ANOVA) comes up a lot in studies.  This topic is in the book.
2. Bayes Rule is important, also in the book.
3. Two-way tables, Simpson's Paradox, are in the book.
4. There is more to be said about tests of significance and confidence intervals.
5. We could also spend time reviewing everything else we have covered.

Or we could do something else.  Something involved with data science.

## Discussion

On Monday, I want to discuss where we are headed.

1. What are your ambitions, goals and dreams?
2. Have your ambitions, goals and dreams changed as a result of last week's election results?
3. As of last week, should we now be more concerned with engaging productively and helpfully in the world?
4. What could we do, as a class, that would promote your ambitions, goals, dreams?
5. What could we do in a few days with basic statistics and/or data science that would help us engage in the world productively and helpfully?  (See below for an idea).
6. If there is time we might consider this question:  What can we do beyond the classroom to engage in the world productively and helpfully?

## My idea
I thought, as one possibility, I would teach you skills for analyzing data from the twitter archive.  Understanding our culture, and being able to convey insights to others, may be a needed skill for the world and our nation, right now.  You could spend a career learning how, but what I plan to do will take two class periods.  After two class periods, we can talk about where to go from there.

I expect that setting up your computers with the free software needed to connect to twitter and create these word clouds will take most of a class period.  You will do some of it at home.  Once you have the software, you will load the code that created this document press and press one button to regenerate this document with the word cloud below.  On the second day we will learn how and where to change the attached file to create a similar document for your own purposes.

## Compassion
_Compassion_ stems from an open heart.  If you feel compassion, you wish for the freedom from suffering of all sentient beings.  **In compassion lies beauty**.  Let us resolve to help open the heart of the world, and heal our suffering.

## Inspirational image.
![Beautiful compassion.](compassion.jpeg)

## Word cloud
Below is a word cloud derived from a sample of 500 recent tweets that include the word "compassion" taken from the twitter archive.  I restricted the search to tweets posted in the month prior to November 8, 2016, election day, and from within 10 miles of the American Univerisity campus.

```{r thewordcloud, results='hide', echo=FALSE, message=FALSE}
##### EDIT SEARCH PARAMETERS BELOW
search_term <- "compassion"  # For two terms use e.g. "compassion+love"
                             # For hashtags use e.g. "#compassion"
language <- "en"
sample_size <- 500
AU_geocode <- '38.9375300,-77.0868600,10mi'  # Latitude, Longitude, Radius
                                             # Here specifies 10 miles from AU campus
since <- '2016-10-08'  # Election Day
until <- '2016-11-08'  # INSERT Until Date, not used here
##### EDIT SEARCH PARAMETERS ABOVE

# NEXT 4 LINES LOAD LIBRARIES OF FUNCTIONS
library(twitteR)
library(tm)
library(wordcloud)
library(RColorBrewer)

# NEXT LINE AUTHENTICATES SESSION WITH TWITTER
source("my_access.R")

####### EDIT THE NEXT LINE TO SPECIFY SEARCH
tweets_raw = searchTwitter(search_term, 
                           n=sample_size,
                           geocode=AU_geocode,
                           since=since,
                           until=until,
                           lang=language)
####### EDIT THE PREVIOUS LINE TO SPECIFY SEARCH

# NEXT TWO LINES REMOVES META-DATA THEN CONVERTS DATA TO A NEEDED FORMAT
tweets_text = sapply(tweets_raw, function(x) x$getText())
tweets_corpus = Corpus(VectorSource(tweets_text))

# NEXT LINE REMOVES SPECIAL CHARACTERS AVAILABLE IN OTHER LANGUAGES
tweets_transformed1 <- tm_map(tweets_corpus, 
                               content_transformer(function(x) iconv(x, to="UTF-8")),
                               mc.cores=1)
# NEXT LINE CONVERTS EVERYTHING TO LOWER CASE
tweets_transformed2 <- tm_map(tweets_transformed1,
                              content_transformer(tolower),
                              mc.cores=1)
# NEXT LINE REMOVES PUNCTUATION
tweets_transformed3 <- tm_map(tweets_transformed2,
                              removePunctuation,
                              mc.cores=1)
# NEXT LINE REMOVES "STOPWORDS" LIKE "THE", "AN", "IS", "BE", ETC.
tweets_transformed4 <- tm_map(tweets_transformed3,
                              function(x) removeWords(x,stopwords()),
                              mc.cores=1)

###### EDIT NEXT LINE TO SPECIFY DRAWING OF WORD CLOUD
wordcloud(tweets_transformed4, random.order=FALSE, colors=brewer.pal(8,"Dark2"))
```

## Code

Attached below is the code used to generate this document.  It contains text, markup, and code for connecting with twitter, downloading a sample of tweets, and creating the word cloud.