Heart rate and sentiment experiment design

Photo GrejGuide.dk @ Flickr

After I finished first experiment and publish article in HEALTHINF 2016 conference this year in Rome I started thinking about next experiment design.

As I mentioned in previous paper improvements I tried to get a lesson from previous mistakes and improve a lot. First, steps are increasing during the day and thus they are not so much independent. Better would be to use heart rate because is totally independent. Second, I can improve my records about sentiment in timing and evaluation. And last but least important is sentiment extraction, instead of supervised learning used in previous work I would like to used unsupervised classification.

So, let's get to details.

Experiment Design

We are still looking for relation between soft data (sentiment) and hard data (measurand). In first experiment it was text recorded via twitter and footsteps. This time it's again text recorded via twitter, but instead footsteps it's heart rate which is more idenpendent.

What's are the main objectives:

• One month experiment (30 days)
• 20 tweets per day (600 tweets minimum)
• Continuous heart rate measurement (24/7)
• Effective time for heart rate measurement between 7 and 23, i. e. 16 hours a day, rest used for wristband charging
• No sleep activity monitoring
• Steps monitoring? Perhaps.

Result of fitness band experiment

As I am progressing with my PhD study I haven't been able to write down any article because I have been busy. And not only PhD made me busy for whole time since January this year, I have to care about work and family.

Nevertheless I should point out results from Introduction fitness band experiment when I wrote about it one and half year ago and never continued.

What was the experiment about

To find relation between sentiment (represented by text recorded in twitter just for practical reasons) and human activity represented by footsteps. Practically about finding link between soft data - sentiment - and hard, measured data.

Reading data: Github implementations

Data and it's processing and analysis code I will publish another time, because I don't have them yet at GitHub.

Twitter data reading implementation

What I have is implementation or reading tweets from Twitter API through tweepy. Which is refactored original version. The reason of refactoring is that I work on second version of fitness band experiment.

Jawbone data reading implementation

What I also have is implementation of reading data from Jawbone API through many different libraries. It's small mess which I need to clean up later when I will use Jawbone again.

Story telling

Long story short

All the data has been extracted, processed and was defined hypothesis which wasn't rejected. Unfortunately, rejection of null hypothesis in favor of alternative was expected and it doesn't happened. That's result and that's the long story short. The whole article presented on HEALTHINF 2016 conference in February this year (2016) is possible to get here:



You can also follow up with the whole story in following chapter, if your are not interested in paper it self right now.

Every Data Scientist must only pay taxes and die, rest is just optional...

Picture from cacm.acm.org

I am just wondering. There are many articles about what every Data Scientist MUST know and do to be real Data Scientist. There are many articles about MUST not do as Data Scientist. What real Data Scientist MUST read and so on and so on. And honestly I don't care. Why is that so?

Research

Let's take the last one must: "What Data Scientist must read" list. I just briefly took several results from Google, here is the list of 10 of them:

• https://www.quora.com/What-is-a-list-of-books-that-every-data-scientist-should-read
• http://bigdata-madesimple.com/20-must-read-books-for-budding-data-scientists
• http://www.analyticsvidhya.com/blog/2015/06/infographic-read-books-analytics-data-science
• http://www.datasciencecentral.com/profiles/blogs/15-books-every-data-scientist-should-read
• http://www.mastersindatascience.org/blog/7-must-read-books-for-the-budding-data-scientist
• http://www.forbes.com/sites/gilpress/2015/03/31/3-recent-books-for-budding-data-scientists-and-business-executives-involved-with-big-data-analytics
• https://www.umbel.com/blog/big-data/must-read-big-data-books
• http://analyticstraining.com/2014/five-must-reads-for-data-scientists
• https://www.quora.com/What-are-the-must-reads-for-data-science-and-machine-learning
• http://www.analyticsvidhya.com/blog/2015/06/infographic-read-books-analytics-data-science

And what is this quick research good for? From this circa 80 books and list of several articles you get list of subjective chosen resources which lead practically to nowhere. Just several books repeating like famous Nate Silvers Signal and Noise and of course some R or other Cookbooks. So, what is conclusion?

Conclusion

This lists of books which someone else read leads me always to Vincent Granville's article Fake data science. And what you need to take from it? Pick any book you need for your field of expertise. And what should be your field of expertise? Choose some project, doesn't matter if your personal one, school or for instance from Kaggle.com. And follow up approaches which you need to for goal achievement, then pick book, course to support your path towards this goal. And by real work, life experience you will sooner or later become Data Scientist.

Bunnies, Dragons and the 'Normal' World: Central Limit Theorem

Statistics is difficult field, but with this video you can see under the hood of one of many things from statistics theory and it's fun.

So, enjoy it as I enjoyed it. And learn something.

Course: Exploratory Data Analysis

What do you imagine under Exploratory Data Analysis?

After a while I decided to continue with one of the first specializations on Coursera which I started more than year ago. This Data Science specialization contains 4th course with the same name as the topic which I was asking in opening question: Exploratory Data Analysis.

I was wondering how much effort I need to include and how much knowledge I can get. After 2 years (yes, anniversary of this blog is 18th of December 2015) with Data Science study path and more than 1 and half year with intensive usage of R language and couple of statistics and R courses. What can surprised me, right?

Nothing and something. I mean this both in positive way and will tell you why.

Computing text conditional entropy with uni- and bi-grams in R and Python

During my first semester of PhD study I have implemented solution for computing conditional entropy over text where each word (including interpunction) was on separate line. I would like to share this example, because it is simple and good for get basics (information theory, R and Python programming).

Conditional entropy is defined as (source is Wikipedia):

Just note, this computation determine the conditional entropy of the word distribution in a text given the previous word.

According line 4 in previous formula, I have to compute p(x,y), which is the probability that at any position in the text you will find the word x followed immediately by the word y, and p(y|x), which is the probability that if word x occurs in the text then word y will follow.

Or according line 5 in formula, I can use probability p(x,y) twice and calculate p(x) which is the probability of single word appearance in the text.

Course: Data Mining Specialization on Coursera

When Specializations started last year on Coursera I joined to Data Science Specialization and was really keen to do as much as I can. I did 3 courses in row so far and then have been stopped with my own activity about PhD study and also little bit disappointed by courses quality which really differ course by course.

Anyway with new year, and I think it was announced during November last year, I joined to first course under Data Mining Specialization - Pattern Discovery in Data Mining. With expectation of better quality than previous courses of Data Science Specialization and more information which I can learn from those courses. It already started from 9th of February, so lets have a more detailed look over it and I will tell you my first impression from it.

Data Mining Specialization is prepared and lead by people of University of Illinois at Urbana-Champaign and contains following courses which you can do in any order (they are independent):

1. Pattern Discovery in Data Mining
2. Text Retrieval and Search Engines
3. Cluster Analysis in Data Mining
4. Text Mining and Analytics
5. Data Visualization

So far, each course have just 1 term offered (Feb 9th, Mar 16th, Apr 27th, Jun 8th, Jul 20th) so you can follow up order above month by month. Either for free or with verification payment of $49 (which is the same as in Data Science Specialization courses) which leads to certificate.

So, I am little bit disappointed that I cannot start with Text Mining, which is most interesting for me and need to wait for it 3 months, but it needs some time to be fully ready, I suppose.

At the beginning of Pattern Discovery course I need to say they were some lacks about organization and course preparation, but it gets better and better and after 2 weeks of study (you have always some days in advance for each week to finalize quiz) it is interesting study, but so far it seems to me, that I am missing some more details, some tough study going deep into problematic. Maybe it is given just by passing the quizzes and missing some implementation assignments.

Anyway I can strongly recommend to you to test it for free (you have couple of weeks before you need to pay for verified certification path) and then decide if you like it or not. Based on my experiences each teacher and each university make different courses. Different approach, presentation, tasks, quizzes and so on, so you need to get used to it.