Developed in partnership with the Independent Schools’ Bursars Association (ISBA) the tool provides financial and operational benchmarking with scenario planning for the sector across the UK and its Crown Dependencies.

Watch the video introduction by John Murphie of ISBA and Allan Engelhardt below:

This intensive course gives you hands-on experience using Microsoft R and Azure covering the key techniques for analysing customer data for Sales and Marketing. The focus is on getting to the business results and you will return to your organization with the skills you need to deliver and demonstrate commercial impact from your work.

For more information and detailed syllabus, please see the course page: Marketing Analytics using Microsoft R and Azure

We are offering the course in partnership with Microsoft (following their acquisition of Revolutions); you can see our listing on Microsoft’s Learn Analytics portal.

Hope to see you there! If you have any questions, just contact us.

Lift curves and the wrong way to use them

This time, the first day of the course was Halloween. But we didn’t have any special spooky data sets or exercises, just the usual programme of an introduction to marketing and analytics in marketing followed by an introduction to set up and evaluate direct marketing campaigns with practical exercises using the R platform for statistical computing and data visualisation.

Lift curves are often used in marketing. You first create a model that predict the probability that a given customer will respond positively to your campaign. A positive response usually means they buy something, but you might have other objectives such as getting more visits to your web site, have your customers refer friends to become your customers, or other outcomes. The model is created by considering the responses from previous campaigns or by comparing test and control groups from a trial. In the course we show how to create the model using generalised linear models (glm), decision trees (rpart, party/partykit), and ensembles of trees, primarily generalised boosted regression models (gbm) but also forests of trees (randomForest) (which we are somehow discouraged from calling Random Forests due to trademark problems). These are of course not all the model types you could consider, but we are limited by the duration of the course and in practise they work well for a wide class of problems. By consistently using the caret package we give the students a scalable toolkit for exploring other model approaches.

Once you have a model that predicts the probability of a positive response, you score your customer base (or the subset eligible for the campaign) and sort the list by the probability from high to low. The cumulative sum of probabilities to n gives you the expected sales from contacting n customers, and the line of this is the lift curve.

The way marketing people tend to use it is that they have a budget for N contacts, for example direct mailings, and they read off the curve how many responses they are going to get. Or if they need M responses (typically sales), they can find the number of contacts needed by starting on M on the y-axis and finding the corresponding N on the x-axis.

Simple.

And wrong.

The right way to do it

Or at least the approach is not optimal in the setting where the product you are selling (or the behaviour you are encouraging) is readily available to all. The problem is that some of the people who buy are people who would buy anyhow, without your marketing effort, so you are wasting some of your budget. You want to target the customers whose behaviour you are most likely to influence. It should not be a surprise that some of the people who purchased during your test campaign, the test campaign you are using to create the model, would have bought from you anyhow.

You have both a treatment and a control group and it seems a real shame not to use the data from both. The treatment group obviously gives you the model for the propensity to buy.

We teach our students to make a baseline model from the control group to predict who would buy without any stimuli. The difference in the probability from the two models is the net response probability and this is the number you want to sort from high to low to decide on your campaign list.

The two lessons are

1. Never throw away good data. In this case, do not ignore the response from the control group.

2. The outcome of marketing activities is to change human behaviour, so make sure you model the change from your efforts and realise that this is not the same as the overall behaviour. You are not the centre of the universe.

Learn more

If you want to learn more, consider signing up for our training course Marketing Analytics Using R or feel free to contact us for an informal conversation.

My background is the hard science of experimental particle physics at CERN where nothing is discovered unless you have replicated the experiment with at least five sigma significance. Max Planck is one of the founders of this science, so when he says that experiment is the only means of knowledge I am with him. It is in my DNA.

The beauty of Marketing is that in many, many cases we can indeed experiment and gain knowledge that way. We go through all the basics both concepts and practical with hands-on exercises on real data in our Marketing Analytics Using R training course.

Consider a direct marketing campaign, that is, a message or a series of messages directed to identified individuals with the objective of changing their behaviour. It could be delivered classically as addressed postal letters, emails, telephone calls, and so on, or as targeted web site promotions or social media campaigns but to identified users. The key is that you know who gets to see the message (at least initially; we’ll leave the effect of sharing and viral campaigns to the course). This is sometimes known as ‘below the line’ (BTL) advertising.

This is different from what we might call advertising, that is broad messages to groups of individuals. You might think classical TV, radio, or newspaper ads, or broad banner ads, google ads, paid-for blog posts, and so on. While the target group may be identified, e.g. by the TV they watch, magazines they read, or web sites they visit, the individual is not known, only the aggregate group characteristics. Old school marketing managers may call this ‘above the line’ (ATL) advertising.

For direct marketing the key concept is that of test and control groups. If you come from a digital marketing background then you may know this as ‘A/B testing’. The basic idea is that you generate your target list of individuals to whom you want to send your message and then take a ramdom sample of that list away as your control group to which you do not communicate. If the two groups are really comparable in terms of the types of individuals in them, and they should be if you selected a true random, unbiased sample, and the only difference in your treatment of the two groups is sending or not sending the marketing message, then any difference in future behaviour, such as different spending patters, must be down to your campaign. This is science, this is experiment, and Planck would be proud. There are some subtleties to make sure you select the groups right (such as no peeking of the results during the experiment) but not much. It is simple, it is effective, and it works.

If you are not using robust test and control groups for your direct marketing campaigns, then start now. Today. Get measuring. Then stop all the campaigns that produce a negative result.

Yes, negative result; campaigns that make customers less likely to buy, more likely to defect to the competition, or more needing of (expensive and unproductive) support from you. We call it Stop Doing Stupid Stuff. At client after client the quick win has more often than not been to Stop Doing Stupid Stuff. Don’t beat yourself up if you have some campaigns with negative returns; I can assure you that you are not alone. It is very easy to become so hung up on your great ideas about what should work that you forget to question if it really does work. Imagination can be useful, but to enhance our understanding of what is real, not instead of it. Just figure out today what real and what is imagines, and then stop the campaigns with negative outcomes. Get the facts.

In our course syllabus we first establish the principles of test and control groups, then we introduce lift curves and consider the difference between gross and net responses before moving on to modelling campaign profitability and optimising your marketing spend: when you have multiple campaigns to run, which one(s) do you execute on which targeted individuals to maximise your profits (or other business objectives)?

“Understanding the reasons for churn, devising the right (effective) churn prevention offers, and measuring (and continue to measure) their effectiveness is usually by far the hardest part of churn management.”

By comparison, the actual churn model is usually quite easy to develop.

Using the same principles we cover churn models to address the business problem of customers leaving you. Some people get confused: what does test and control groups have to do with churn? After all, you don’t choose who will leave you. But then you are doing churn modelling wrong. You typically do not want to predict churn, as in the probability that an individual will leave you. Say you did do this model, and say it was a great model. Now what are you going to do? The model does not directly address the possible business actions or the decisions that the business need to make. Typically, what you want to do is to offer some sort of incentive to say to some of the customers who are likely to leave. So the decision is: for each individual customer, given the range of possible retention incentives that I have at my disposal, would offering one of these incentives at this time to this customer increase my overall profitability? You might frame this question as ‘what is the expected change in customer lifetime value from presenting each of these offers at this time?’ which is something that you can usefully model. You need the cost of the offers, the propensity to accept it, the baseline customer lifetime value, and the change in spend and tenure or loyalty given the offer.

Sure, it is more complicated, but it is useful in a way that the simple ‘Will this customer churn?’ model is not. We wrote more on commercial churn modelling and we cover it in practical detail in the Marketing Analytics Using R training course.

Our philosophy is simple. The focus is always, or should be, on the business actions. Models do not make money, changing the way you do business may. Start with the end in mind, measure the right outcomes, and directly model the required business decisions, and you should be fine. But it is easy to get absorbed in the data and modelling and we have plenty of examples of that. Business understanding is key to successful Marketing Analytics.

We do not usually go into details on the statistics. Typically in the problems with which we deal in Marketing Analytics, we have thousands if not millions of records. Thousands of customers, tens or hundreds of thousands of purchases or transactions, or millions of web site visits. A signal from thousands of observations will usually be statistically significant, at least in the naive sense, and with an emphasis on experimentation that is normally good enough. So we spend more time on feature selection and model optimisation.

Feature selection is important on the wide data sets that are typical of our problems, and creating the right (derived) features is very often the key to developing a successful model. The small response rates and wide data sets means that we often use modern modelling techniques that are extremely powerful but may make interpretation more difficult. We compare simple and complex models in the course and show how to optimise feature selection and model performance using the caret package and framework. (The course instructor is an acknowledged contributor to the caret package.)

Marketing Analytics Using R training course

Learn more about our Marketing Analytics Using R training course. You can also Contact us to register your interest and receive more information on this course, and we will let you know the next time we run a public class. If you have several colleagues who are interested then we can also run a course just for you.

That has now changed. We are pleased to announce new dates for this popular course with the next training class scheduled for 31 October - 04 November 2016 at a time that could suit participants from Dubai to San Fransisco.

The format of the course is 5 half-day live tutorials delivered using an online classroom platform. This format is chosen so students still have time to do their day-jobs. This means 20 hours of face-time with the teacher plus the opportunity to dive deeper into the technical aspects through homework exercises for which the instructor is available by email or chat.

Register here: Event Registration. Note that there is a generous early bird discount available.

Read more about the course including the outline syllabus here: Marketing Analytics Using R training course

Announcement with some background: LinkedIn announcement

And if you are reading this after this course has finished, contact us to register your interest and be on the mailing list when a new schedule is announced.

In 2015 Microsoft acquired Revolution Analytics (MSFT, Revo) which gave them a serious and scalable tool for data science, visualization, and statistical analysis. This is based on the open source R implementation which has massive mind-share making it easily the most popular serious data science tool, but adds easy scalability and enterprise features.

And they announced integration with Apache Spark (Microsoft Azure HDInsight), Azure Machine Learning, and SQL Server, which suddenly looks like a serious platform. Add a 75% reduction in the price of Power BI (and a reasonable free version) and various other enhancements, and it is looking really interesting.

So is it ready for prime-time? I had the opportunity this week to spend four days with the Microsoft teams to find out. We focused on the R environments and integrations and received plenty of hands-on experience.

Solutions components

We covered the following solutions components:

1. The scalable rx* model and data processing functions using the XDF file format.
2. The integration with the dplyr package through dplyrXdf for streaming data manipulation.
3. The Apache Spark / HDInsight integration: using essentially the same syntax, but automatically distributed over a Spark cluster.
4. The Microsoft SQL Server integration: Using the same rx* functions against database tables either from the client or running on the SQL Server.
5. Running R inside the SQL Server using stored procedures (and sp_execute_external_script).

1. Scalable model and data processing functions

This is the venerable technology from Revolutions. It provides functions that work transparently on chunks of data which means you have no issues of memory limitations. For data processing the key function is rxDataStep which does all the transformations you would expect and is supported by rxMerge for merging data and rxImport for importing data files. This all works well and was the core offering from Revolutions, along with the XDF file storage format which allows efficient processing of large data.

For data exploration you have comprehensive summaries (including approximate median), contingency tables, and cross tabulations, and there is a small but useful selection of data visualization functions. Models are currently limited to

• Linear Regression Models (rxLinMod)
• Logistic Regression (rxLogit)
• Generalized Linear Models (rxGlm)
• Classification and Regression Trees (rxDTree)
• Stochastic Gradient Boosted Decision Trees (rxBTrees)
• Classification and Regression Decision Forests (rxDForest)
• K-Means Clustering (rxKmeans)
• Naive Bayes Classifier (rxNaiveBayes)

More model types are apparently in the works. I hope to see association rules (the Apriori and Eclat algorithms at least) and anomaly detection (eg one-class support vector machines) at least. I hear rumours that support vector machines (SVM) are coming soon, and of course they provide the infrastructure for you to implement your own models.

Nothing new here compared to the Revolutions offering when you are working on XDF files. What is new is the seamless integration with Spark clusters meaning that you can easily run any of these models on an efficient distributed environment.

The code we went through is here: MRS for SAS Users

2. Streaming data manipulation the dplyr way

This is the dplyrXdf package which was announced about three months ago. At the moment you have to install it from the dplyrXdf GitHub page.

I was skeptical at the start but quickly convinced. If you want to quickly compare the syntax you can see the course material here. So you get the dplyr syntax, that’s nice. But you get it distributed on a Spark cluster with (almost) no change of code: that’s awesome! (And you don’t have to keep track of your temporary files: that is sometimes useful.)

3. Microsoft R with Spark on HDInsight

It is easy to set up (though not quite as easy as Amazon EMR yet); the instructions are here but note that this is rapidly changing. The quick summary is that it works really well. The key function here, and for the SQL Server integration, is rxSetComputeContext which is a general function that defines where your computation is taking place. On Spark you have many options but you end up with code along the lines of:

computeContext <- RxSpark(consoleOutput = TRUE,
                          nameNode = myNameNode,
                          port = myPort,
                          executorCores = 6, 
                          executorMem = "3g", 
                          executorOverheadMem = "3g", 
                          persistentRun = TRUE, 
                          extraSparkConfig = "--conf spark.speculation=true")

rxSetComputeContext(computeContext)

linmod <- rxLinMod(tip_pct ~ trip_distance, 
                   data = taxi_xdf, blocksPerRead = 2)

There is some good documentation here: here: Overview of R Server on HDInsight

Bottom line: it is easy to use, it works, it scales. But you still have to write good R code, of course.

4. Microsoft R and SQL Server

For an overview and to really understand the workflow for this and the following component, this document is the best: Data Science End-to-End Walkthrough.

But briefly you can work in two modes. The first is a transparent use of SQL tables as yet another XDF compatible data source, all processed in chunks and on your local machine. Works well, with the obvious limitations of data type compatibility etc. The downside is that all the data has to move from the server to the R instance. But the second mode will fix that: with a suitable invocation of our old friend rxSetComputeContext you can move the computation to the database server.

At the moment, at least, the compute context interface is rather clunky and you end up with lots of local files that created as part of the communications with the server. (See the shareDir={:.language-r} argument of the RxInSqlServer function.) This is ugly and error-prone.

Currently this only really works with the (Enterprise) SQL Server and not, for example, with the Azure SQL service. Very much work in progress, but give it time.

5. Running R as stored procedure in SQL Server

For its final trick, SQL Server is able to run R directly in the server as a stored procedure or similar. The key function is sp_execute_external_script which you can include in stored procedures and similar.

In a typical use case, the analyst builds the model using the rx* model functions described above using one of the standard user interfaces and appropriate compute context. She then stores the (serialized) model in the database and the DBA sets up a stored procedure to calculate the scores for new (eg in a trigger) or changed (eg scheduling a stored procedure) data, or when the model changes. In other words, this is mainly a deployment option. It works well.

Conclusions

There is much to like here. Microsoft is building an infrastructure and services platform that is robust and scalable, and with a good or great partner ecosystem. They are one of the most interesting technology companies in this space, and I am not the only one who is excited for their vision; Gartner says something similar.

Who is it for? This is a technical platform for technically savvy data scientists. If you want something easy to use, go to Oracle Data Miner where everything is simple and works almost as if by magic, but you sacrifice some control for this convenience. The Oracle set of parallel algorithms is more comprehensive than the rx* models described above which in the current implementation in particular lack association rules and integration into text mining, and the integration to R in the Oracle database appears more mature and complete.

But Oracle does not have a simple and consistent interface that works across files, databases, and Spark clusters. And in particular they do not have anything really comparable to Azure and the ecosystem of integrated services you find there to help you build your solution fast and cost-effective. In that sense the nearest competitor is Amazon Web Services which has even more integrated services, many of which are more mature. (For example, it is easy to have R Studio Server pre-configured on the EMR Spark service at AWS but a bit of a hack is needed on Azure.) But it does not have the same integration with Microsoft R Server for the large data set models so you will have some more work to use that.

Microsoft is by no means the only play in town, but they are certainly credible right now for your data and data science infrastructure and they are an interesting company to watch.

The Net Promoter Score (NPS) is a metric for measuring the loyalty and engagement of a firm’s customers that is extremely popular in across many industries. Originally introduced by Fred Reichheld in a 2003 article in the Harvard Business Review (link); the current reference is the book The Ultimate Question 2.0 by Reichheld and Markey (Amazon paid link UK | US).

The score is calculated by asking the firm’s customers a single question, How likely is it that you would recommend our company/product/service to a friend or colleague?, on a scale from 0 (very unlikely) to 10 (very likely). Those customers who respond 9 or 10 are considered Promoters, those who score 7 or 8 are Passives, and the rest scoring 0-6 are Detractors. The Net Promoter Score (NPS) is then simply the percentage Promoters minus the percentage Detractors, which gives a number between -100 and +100.

Our concern here is estimating the measurement error of the score given the responses to the survey.

Maths and Stats: Closed form for the variance

Let’s start by deriving the closed form for the variance of the Net Promoter Score. Skip this one if you only want the code.

Let \( n \) be the number of respondents to our survey and let \( p \) and \( d \) be the proportion of respondents who are Promoters and Detractors, respectively. Then

and by the usual formula for variance we have

\[ \Var(NPS) = 100^2 \times ( \Var(p) + \Var(d) - 2 \times \Cov(p, d) ) \]

where \( \Cov \) denotes the covariance.

We can find the variance and covariance from the simple sample and the Categorical Distribution so \( \Var(p) = p \times (1-p) / n \) and \( \Cov(p, d) = {-p} d / n \) where we assume the total population \( N \) of possible respondents is much greater than \( n \). We then get

where \( \sd \) denotes the standard deviation.

For the special case \( p = d = ⅓ \) (which is one case of a zero NPS) we considered in the previous post and using the normal approximation for the 95% confidence interval \( \CI \approx \pm 1.96\sd \), we get for the full width of the interval

For a full width confidence interval of 10 or 1, you need 1024 or 102,400 responses, respectively, which gives us the rule of thumb we mentioned in the previous post:

For the very impatient, a good rule of thumb is that with 1000 responses you (only) know your Net Promoter Score to within 10 points (±5) and it takes fully 100,000 responses to know the score to one point (±0.5).

Practical implementation using R

Knowing the formula for the variance is all well and good but we are data science people so let’s work with the data to create the confidence intervals from the data in a way that works even when the score is near 100 and which extends to other values you may want to derive from the data. In the code examples below we use the R software environment for statistical computing and graphics but it should be easy to translate to other languages.

To create our confidence intervals we use resampling techniques, specifically the bootstrap method as implemented in the boot library. The code below implements a simple function for bootstrapping the errors.

library("boot")
## Function for bootstrapping the sample error estimates
NPS.boot <- function(x, R = 1e4, conf = 0.95, type = "basic") {
  NPS <- function(x) 100*(sum(x >= 9) - sum(x <= 6))/length(x)
  b   <- boot(x, function(data, i) NPS(data[i]), R = R)
  ci  <- boot.ci(b, conf = conf, type = type)
  res <- as.list(c(NPS(x), length(x), ci[[type]][1, c(1, 4:5)]))
  names(res) <- c("NPS", "n", "conf", "Low", "High")
  return(res)
}

In the code, x is a vector of responses to the key NPS question (Likelihood to recommend), R is the number of resamples you want to do, and conf is the confidence interval. We take the confidence interval from the observed quantiles of the distribution as indicated by type = "basic"; see the boot.ci help for other options but this choice is our recommendation.

Then we can use the function simply as NPS.boot(nps$Likelihood.to.Recommend) if nps is our data of survey responses and Likelihood.to.Recommend is the integer 0-10 value from the response to the NPS question. Or, more likely, we can use it to generate confidence intervals for a range of periods, eg months, quarters, or years, as outlined below.

## Note that we use the data.table syntax below.
library("data.table")
## Assume nps holds your survey data, one row per response where
## - Likelihood.to.Recommend is the 0-10 score which we assume is not NA here
## - quarter is the time period over which you want to aggregate, here quarterly
nps[, NPS.boot(Likelihood.to.Recommend), by = quarter]

To try out the code above, we can generate some data

## Note that we use the data.table syntax and its IDate class below.
library("data.table")
set.seed(20160115)

n <- 12e3  # Generate this many responses
p <- c(rep(1,7), 3, 4, 4, 3) # Response probabilities; true NPS = 0

nps <-
  data.table(Likelihood.to.Recommend =
               sample(0L:10L, n, replace = TRUE, prob = p),
             Response.Date =
               sample(seq(as.IDate("2013-01-01"),
                          as.IDate("2015-12-31"),
                          by = "day"),
                      n, replace = TRUE))

# Add column `quarter`
nps[, quarter := round(Response.Date, digits = "quarters")]
setkey(nps, quarter)

qs <- nps[, NPS.boot(Likelihood.to.Recommend), by = quarter]
print(qs)

       quarter        NPS    n conf         Low      High
 1: 2013-01-01  0.3211991  934 0.95 -4.81798715  5.674518
 2: 2013-04-01 -0.3036437  988 0.95 -5.36437247  4.655870
 3: 2013-07-01  2.1632252 1017 0.95 -2.85152409  7.079646
 4: 2013-10-01 -0.4935834 1013 0.95 -5.42941757  4.343534
 5: 2014-01-01 -2.9927761  969 0.95 -8.25593395  2.270382
 6: 2014-04-01  0.3992016 1002 0.95 -4.79041916  5.489022
 7: 2014-07-01  1.7329256  981 0.95 -3.46585117  6.931702
 8: 2014-10-01  5.1536174 1009 0.95  0.09910803 10.208127
 9: 2015-01-01 -2.6679842 1012 0.95 -7.80632411  2.371542
10: 2015-04-01 -3.4644195 1068 0.95 -8.33333333  1.404494
11: 2015-07-01  0.1021450  979 0.95 -5.20939734  5.209397
12: 2015-10-01  1.0700389 1028 0.95 -3.98832685  6.031128

Note that, even with 1000 responses per quarter, one of the 95% confidence intervals (number 8) exclude the true value of NPS=0; with 12 periods that is quite reasonable (probability 46%). Note also that we have an average of 1000 responses per quarter and the full width of our confidence intervals are indeed around 10, as per our rule of thumb.

For a simple plot of the data we could use

library("ggplot2")

qs.plot <-
  ggplot(qs, aes(x = quarter, y = NPS, ymin = Low, ymax = High)) +
  scale_x_date() +
  geom_hline(aes(yintercept = 0), linetype = "dashed", colour = "red") +
  geom_point() + geom_errorbar(width = 30) +
  ggtitle(expression(atop("Quarterly NPS",
                          atop(italic("with 95% confidence intervals"), ""))))

plot(qs.plot)

Note that there is no true trend in this data. If you think you see a pattern where the first two quarters of the year are low and the second two are high then you are just imagining it. If you think your score fell in Q1 2015 but is now on the mends since Q3, you are just imagining it.

The approach extends trivially to other metrics you want to calculate from the survey data. One of our automotive clients tracked the proportion of ‘negative’ responses 0-4 to the Likelihood to recommend question; negative in the sense that the customer had selected below the middle (5) on the 0-10 scale.

We can do that easily

library("boot")
library("data.table")
library("ggplot2")
## Function for bootstrapping the sample error estimates
negative.boot <- function(x, R = 1e4, conf = 0.95, type = "basic") {
  neg <- function(x) 100*sum(x <= 4)/length(x)
  b   <- boot(x, function(data, i) neg(data[i]), R = R)
  ci  <- boot.ci(b, conf = conf, type = type)
  res <- as.list(c(neg(x), length(x), ci[[type]][1, c(1, 4:5)]))
  names(res) <- c("Negative", "n", "conf", "Low", "High")
  return(res)
}

set.seed(1)   # Reproducible bootstrap values

nqs <- nps[, negative.boot(Likelihood.to.Recommend), by = quarter]

nqs.plot <-
  ggplot(nqs, aes(x = quarter, y = Negative, ymin = Low, ymax = High)) +
  scale_x_date() +
  geom_hline(aes(yintercept = 500/21), linetype = "dashed", colour = "red") +
  geom_point() +
  geom_errorbar(width = 30) +
  ggtitle(expression(atop("Proportion of Negative responses",
                       atop(italic("with 95% confidence intervals"), ""))))

plot(nqs.plot)

This gives us:

If you have, say, a threshold of 25% for this metric and you only use the mean from the data, you would have had 3 false positives in this three months period. Or if your threshold was 22.5% you would have had 1 false negative.

Conclusion

I hope to see many more plots of NPS with error bars. As we mentioned previously, the score is not really the core of the system; closing the loop with the customers, understanding the root causes, and taking appropriate action is the core. Nevertheless, it is easy for an organization to become somewhat fixated on the score which leads to demands to ‘explain’ every change in the measured value.

Hopefully this analysis will allow the practitioner to have an informed conversation about the magnitude of the change and whether it may just be down to sampling. But note that we do not advocate ignoring the ‘why’ question even when sample sizes are small and the change is not statistically significant: it is extremely valuable to be collecting hypotheses for what may affect the score and try to test these. That too is part of closing the loop.

Feel free to contact us with any questions or comments.

The intended audience of the conference includes both academics and practitioners who are active or interested in the applications of R in Insurance.

Attendance of the whole conference is the equivalent of 6.5 hours of CPD for members of the Actuarial Profession.

Sign up here: R in Insurance 2016: Registration and Fees.

At this stage, our purpose is to reproduce the analysis from the book using the R statistical computing and analysis platform, and to answer the data analysis elements of the exercises and case studies. Any critique of the approach and of pricing and modeling in the Insurance industry in general will wait for a later article.

In the following, we will assume that the reader has a copy of the book and a working installation of R. We will be using the data.table, foreach, and ggplot2 packages which are not part of the standard distribution, so the reader should install them first (e.g. by executing the following line from within an R session:

install.packages(c("data.table", "foreach", "ggplot2"), dependencies = TRUE)

The beginning of the code is not so important, but here goes:

#!/usr/bin/Rscript
## PricingGLM-2.r - Code for Chapter 2 of "Non-Life Insurance Pricing with GLM"
## Copyright © 2012 CYBAEA Limited (http://www.cybaea.net)

## @book{ohlsson2010non,
##   title={Non-Life Insurance Pricing with Generalized Linear Models},
##   author={Ohlsson, E. and Johansson, B.},
##   isbn={9783642107900},
##   series={Eaa Series: Textbook},
##   url={http://books.google.com/books?id=l4rjeflJ\_bIC},
##   year={2010},
##   publisher={Springer Verlag}
## }

Example 2.5: Moped insurance continued

We continue the moped insurance example, and we use the data that we saved in our Chapter 1 session. The goal is to reproduce Table 2.7 so we start building that as a data frame after loading the data.

## Load the data from last.
if (!exists("table.1.2"))
    load("table.1.2.RData")

library("foreach")

## We are looking to reproduce table 2.7 which we start building here,
## adding columns as we go.
table.2.7 <-
    data.frame(rating.factor =
               c(rep("Vehicle class", nlevels(table.1.2$premiekl)),
                 rep("Vehicle age",   nlevels(table.1.2$moptva)),
                 rep("Zone",          nlevels(table.1.2$zon))),
               class =
               c(levels(table.1.2$premiekl),
                 levels(table.1.2$moptva),
                 levels(table.1.2$zon)),
               stringsAsFactors = FALSE)

We next calculate the duration and number of claims for each level of each rating factor. We also set the contrasts for the levels, using the same idiom as in our Chapter 1 session. The {foreach} package is convenient to use here, but you can of course do it with a normal loop and a couple of variables.

## Calculate duration per rating factor level and also set the
## contrasts (using the same idiom as in the code for the previous
## chapter). We use foreach here to execute the loop both for its
## side-effect (setting the contrasts) and to accumulate the sums.
new.cols <-
    foreach (rating.factor = c("premiekl", "moptva", "zon"),
             .combine = rbind) %do%
{
    nclaims <- tapply(table.1.2$antskad, table.1.2[[rating.factor]], sum)
    sums <- tapply(table.1.2$dur, table.1.2[[rating.factor]], sum)
    n.levels <- nlevels(table.1.2[[rating.factor]])
    contrasts(table.1.2[[rating.factor]]) <-
        contr.treatment(n.levels)[rank(-sums, ties.method = "first"), ]
    data.frame(duration = sums, n.claims = nclaims)
}
table.2.7 <- cbind(table.2.7, new.cols)
rm(new.cols)

Next, we need to build the frequency and severity models separately, as in the discussion at the beginning of section 2.3.4 on page 34.

Frequency model

Note here the use of the offset() term as opposed to the weights= argument we have used before. The offset is log(dur) because our link function is log.

See help("Insurance", package = "MASS") for a similar example and sections 7.1 (p. 189–190) and 7.3 of Modern Applied Statistics with S (Amazon paid links UK | US for a (very brief) discussion.

model.frequency <-
    glm(antskad ~ premiekl + moptva + zon + offset(log(dur)),
        data = table.1.2, family = poisson)

rels <- coef( model.frequency )
rels <- exp( rels[1] + rels[-1] ) / exp( rels[1] )
table.2.7$rels.frequency <-
    c(c(1, rels[1])[rank(-table.2.7$duration[1:2], ties.method = "first")],
      c(1, rels[2])[rank(-table.2.7$duration[3:4], ties.method = "first")],
      c(1, rels[3:8])[rank(-table.2.7$duration[5:11], ties.method = "first")])

Severity model

There are a couple of points to note here:

1. We will model using a Gamma distribution for the errors, following the discussion on page 20 and also pages 33-34. Note the point that this is only one of several plausible candidate distributions.
2. Because we are using the Gamma distribution we need to remove the zero values from the data; we do this using the table.1.2[table.1.2$medskad > 0, ] construct.
3. To reproduce the values from the book, we use the non-canonical “log” link function even though the canonical function (“inverse”) gives a slightly better fit (residual deviance 5.9 versus 8.0 on 16 degrees of freedom). This follows the approach discussed in Example 2.3 on page 30.

model.severity <-
    glm(medskad ~ premiekl + moptva + zon,
        data = table.1.2[table.1.2$medskad > 0, ],
        family = Gamma("log"), weights = antskad)

rels <- coef( model.severity )
rels <- exp( rels[1] + rels[-1] ) / exp( rels[1] )
## Aside: For the canonical link function use
## rels <- rels[1] / (rels[1] + rels[-1])

table.2.7$rels.severity <-
    c(c(1, rels[1])[rank(-table.2.7$duration[1:2], ties.method = "first")],
      c(1, rels[2])[rank(-table.2.7$duration[3:4], ties.method = "first")],
      c(1, rels[3:8])[rank(-table.2.7$duration[5:11], ties.method = "first")])

Combining the models

Now it is trivial to combine and display the results.

table.2.7$rels.pure.premium <- with(table.2.7, rels.frequency * rels.severity)
print(table.2.7, digits = 2)

Case Study: Motorcycle Insurance

The authors use the term Case Study for larger exercises. There is quite a lot to cover here, and some of the questions are more about discussing the results. We will focus on getting the results.

The data

Obtaining the data

First we read the (fixed width) data. The column names are the same (based on Swedish language) as in the book.

columns <- c(agarald = 2L, kon = 1L, zon = 1L, mcklass = 1L, fordald = 2L,
             bonuskl = 1L, duration = 8L, antskad = 4L, skadkost = 8L)
column.classes <- c("integer", rep("factor", 3), "integer",
                    "factor", "numeric", rep("integer", 2))
stopifnot(length(columns) == length(column.classes))
con <- url("http://www2.math.su.se/~esbj/GLMbook/mccase.txt")
mccase <- read.fwf(con, widths = columns, header = FALSE,
                   col.names = names(columns),
                   colClasses = column.classes,
                   na.strings = NULL, comment.char = "")
try(close(con), silent = TRUE)
rm(columns, column.classes, con)
mccase$mcklass <- ordered(mccase$mcklass)
mccase$bonuskl <- ordered(mccase$bonuskl)

Adding meta-data information

We are more than a little obsessed with documenting our data, so here goes. See the book for the details.

comment(mccase) <-
    c("Title: Partial casco insurance for motorcycles from Wasa, 1994--1998",
      "Source: http://www2.math.su.se/~esbj/GLMbook/mccase.txt",
      "Copyright: http://www2.math.su.se/~esbj/GLMbook/")
comment(mccase$agarald) <-
    c("The owner's age, between 0 and 99",
      "Name: Age of Owner")
comment(mccase$kon) <-
    c("Name: Gender of Owner",
      "Code: M=Male",
      "Code: K=Female")
comment(mccase$zon) <-
    c("Name: Geographic Zone",
      "Code: 1=Central and semi-central parts of Sweden's three largest cities",
      "Code: 2=suburbs and middle-sized towns",
      "Code: 3=Lesser towns, except those in 5 or 7",
      "Code: 4=Small towns and countryside, except 5--7",
      "Code: 5=Northern towns",
      "Code: 6=Northern countryside",
      "Code: 7=Gotland (Sweden's largest island)")
comment(mccase$mcklass) <-
    c("Name: MC Class",
      "Description: A classification by the EV ratio, defined as (engine power in kW × 100) / (vehicle weight in kg + 75), rounded to the nearest lower integer.",
      "Code: 1=EV ratio <= 5",
      "Code: 2=EV ratio 6--8",
      "Code: 3=EV ratio 9--12",
      "Code: 4=EV ratio 13--15",
      "Code: 5=EV ratio 16--19",
      "Code: 6=EV ratio 20--24",
      "Code: 7=EV ratio >= 25")
comment(mccase$fordald) <-
    c("Vehicle age, between 0 and 99",
      "Name: Vehicle Age")
comment(mccase$bonuskl) <-
    c("Name: Bonus Class",
      "Description: A driver starts with bonus class 1; for each claim-free year the bonus class is increased by 1. After the first claim the bonus is decreased by 2; the driver cannot return to class 7 with less than 6 consecutive claim free years.")
comment(mccase$duration) <-
    c("Name: Duration",
      "Comment: The number of policy years",
      "Unit: year")
comment(mccase$antskad) <-
    c("Name: Number of Claims")
comment(mccase$skadkost) <-
    c("Name: Cost of Claims",
      "Unit: SEK")

Rating factors

We finally add the rating factors from the book.

mccase$rating.1 <- mccase$zon
mccase$rating.2 <- mccase$mcklass
mccase$rating.3 <-
    cut(mccase$fordald, breaks = c(0, 1, 4, 99),
        labels = as.character(1:3), include.lowest = TRUE,
        ordered_result = TRUE)
mccase$rating.4 <- ordered(mccase$bonuskl) # Drop comments
levels(mccase$rating.4) <-                 # Combine levels
    c("1", "1", "2", "2", rep("3", 3))

Save the data

Never forget to save.

save(mccase, file = "mccase.RData")

Now we can start tackling the exercises.

Problem 1: Aggregate to cells of current tariff

Aggregate the data to the cells of the current tariff. Compute the empirical claim frequency and severity at this level.

We really like the data.table package. The syntax is much easier on the eye (and the hand). As a bonus, it scales well to much larger data sets than data.frame. If you are not already using it, now is the time to start.

if (!exists("mccase"))
    load("mccase.RData")

## Conver to data.table
library("data.table")
mccase <- data.table(mccase, key = paste("rating", 1:4, sep = "."))

## Aggregate to levels of current rating factors
mccase.current <-
    mccase[,
           list(duration = sum(duration),
                antskad = sum(antskad),
                skadkost = sum(skadkost),
                num.policies = .N),
           by = key(mccase)]             

## Claim frequency and severity. Change NaN to NA.
mccase.current$claim.freq <-
    with(mccase.current, ifelse(duration != 0, antskad / duration, NA_real_))
mccase.current$severity <-
    with(mccase.current, ifelse(antskad != 0, skadkost / antskad, NA_real_))

## Save
save(mccase.current, file = "mccase.current.RData")

Problem 2: Determine how the duration and number of claims is distributed

Determine how the duration and number of claims is distributed for each of the rating factor classes, as an indication of the accuracy of the statistical analysis.

This is one of those ‘there is no right answer’ questions, but let us have a look at the data. First load it if needed and also load the libraries we will be using.

## Load data if needed
library("data.table")
if (!exists("mccase"))
    load("mccase.RData")
if (!exists("mccase.current"))
    load("mccase.current.RData")
if (!is(mccase, "data.table"))
    mccase <- data.table(mccase, key = paste("rating", 1:4, sep = "."))

library("grid")
library("ggplot2")

1. Number of claims (antskad)

Let us start with the number of claims for the undelying data. First we plot the number of policies for each number of claims (range of number of claims is {0, 1, 2}); this is one way to do it:

plot.titles <- c("Geo. zone", "MC class", "Vehicle age", "Bonus class")
plots <-
    lapply(1:4,
           function(i)
           ggplot(mccase, aes(antskad))
           + geom_histogram(aes(weight = duration))
           + scale_x_discrete(limits = c(0, 2))
           + scale_y_log10()
           + facet_grid(paste("rating.", i, " ~ .", sep = ""),
                        scales = "fixed")
           ## We drop the axis titles to make more room for the data
           + opts(axis.title.x = theme_blank(), axis.title.y = theme_blank(),
                  axis.text.x = theme_blank(),  axis.text.y = theme_blank(),
                  axis.title.y = theme_blank(), axis.ticks = theme_blank(),
                  title = plot.titles[i])
           )

grid.newpage()
pushViewport(viewport(layout = grid.layout(nrow = 1, ncol = 4)))
## We can ignore the warnings from displaying the plots for now
for (i in 1:4)
    print(plots[[i]], vp = viewport(layout.pos.col = i))

There is certainly some variation there, especially on the first rating factor.

Recall from the discussion on page 18 the assumption that the number of claims for a single policy is a Poisson process and the distribution of the number of claims in a tariff cell is Poisson distributed. Let us first look at the whole data set (and note that the data.table notation is used; if using data.frame be sure to have drop=FALSE):

data <- mccase[order(antskad), list(N = .N, w = sum(duration)), by = antskad]
M <- glm(N ~ antskad, family = poisson(), weights = w, data = data)
data$predicted <- round(predict(M, data[, list(antskad)], type = "response"))
print(data[, list(antskad, N, predicted)], digits = 1)

Not too bad, but then it can’t really be with that heavy weighting to the first value of antskad. We can show the same split by the levels of the first rating factor as an example:

data <-
    mccase[order(rating.1, antskad),
           list(N = .N, w = sum(duration)),
           by = list(rating.1, antskad)]
modelPoisson <- function(data) {
    M <- glm(N ~ antskad, family = poisson(), weights = w, data = data)
    return(M)
}
predictionPoisson <- function (data) {
    M <- modelPoisson(data)
    p <- predict(M, data[, list(antskad)], type = "response")
    return(p)
}
data$predicted <-
    unlist(lapply(levels(data$rating.1),
                  function (l) round(predictionPoisson(data[rating.1 == l]))))
print(data[, list(rating.1, antskad, N, predicted)], digits = 1)

You get the idea….

If we can’t really see very much at the individual claims level, how does it appear when we look at the levels aggregated to the current tariff cells? Here, of course, we have much more data with up to 33 claims in one cell, but we will limit the display to the first few number of claims.

high.limit <- 7L                 # Only display up to this many claims
plots <-
    lapply(1:4,
           function(i)
           ggplot(mccase.current, aes(antskad))
           + geom_histogram(breaks = 0:high.limit, aes(weight = duration))
           + scale_x_discrete(limits = c(0, high.limit), breaks = 0:high.limit)
           ## Following xlim() needed for scale_x_discrete only; see
           ## https://groups.google.com/d/msg/ggplot2/wLWGCUz8K6k/DeVudyfXyKgJ
           + xlim(0, high.limit)
           + scale_y_log10()
           + facet_grid(paste("rating.", i, " ~ .", sep = ""),
                        scales = "fixed")
           ## We drop the axis titles to make more room for the data
           + opts(axis.title.x = theme_blank(), axis.title.y = theme_blank(),
                  axis.text.x = theme_blank(),  axis.text.y = theme_blank(),
                  axis.title.y = theme_blank(), axis.ticks = theme_blank(),
                  title = plot.titles[i])
           )

grid.newpage()
pushViewport(viewport(layout = grid.layout(nrow = 1, ncol = 4)))
for (i in 1:4)
    print(plots[[i]], vp = viewport(layout.pos.col = i))

There is certainly something going on for low bonus classes and high vehicle ages that is not straightforward Poisson distribution. Looking at the two in combination we see the old motorcycle - low bonus class (rating.3 = 3 and rating.4 = 1) is the main culpit, but also note that there is relatively little data in these cells.

ggplot(mccase.current, aes(antskad)) +
    geom_histogram(breaks = 0:high.limit, aes(weight = duration)) +
    scale_x_discrete(limits = c(0, high.limit), breaks = 0:high.limit) +
    xlim(0, high.limit) +
    scale_y_log10() +
    facet_grid(rating.4 ~ rating.3, scales = "fixed", labeller = "label_both")

We can fit the poisson distribution using the same predictionPoisson() function as before. We also look at the fit in a simple way (the - much! - better approach is to use the residuals from the fitted model)

## rating.3 is the vehicle age.
data <-
    mccase.current[order(rating.3, antskad),
                   list(N = .N, w = sum(duration)),
                   by = list(rating.3, antskad)]
data$predicted <-
    unlist(lapply(levels(data$rating.3),
                  function (l) round(predictionPoisson(data[rating.3 == l]))))
## Show the fit
print(data[antskad <= high.limit,
           list(rating.3, antskad, N, predicted)], digits = 1)
## Show simplistic residuals per level of the rating factor
print(data[antskad <= high.limit,
           list(res = sum(abs(predicted - N))),
           by = rating.3])

## rating.4 is the bonus class
data <-
    mccase.current[order(rating.4, antskad),
                   list(N = .N, w = sum(duration)),
                   by = list(rating.4, antskad)]
data$predicted <-
    unlist(lapply(levels(data$rating.4),
                  function (l) round(predictionPoisson(data[rating.4 == l]))))
## Show the fit
print(data[antskad <= high.limit,
           list(rating.4, antskad, N, predicted)], digits = 1)
## Show simplistic residuals per rating factor
print(data[antskad <= high.limit,
           list(res = sum(abs(predicted - N))),
           by = rating.4])

We will not show the tables here (as we said: there is a better way). Instead, we can show the fitted Poisson distribution:

ggplot(data, aes(x = antskad, y = N)) +
    geom_bar(breaks = 0L:high.limit, stat = "identity") +
    facet_wrap( ~ rating.4) +
    geom_line(aes(y = predicted), data = data, colour = "red", size = 1) +
    xlim(-0.5, high.limit + 0.5)

2. Duration

We can plot the distribution of duration as before. Here 440 is the 90% quantile for the duration (412) rounded up to the next multiplier of binwidth.

plots <-
    lapply(1:4,
           function(i)
           ggplot(mccase.current, aes(duration))
           + geom_histogram(binwidth = 40)
           + xlim(0, 440)
           + scale_y_log10()
           + facet_grid(paste("rating.", i, " ~ .", sep = ""),
                        scales = "fixed")
           ## We drop the axis titles to make more room for the data
           + opts(axis.title.x = theme_blank(), axis.title.y = theme_blank(),
                  axis.text.x = theme_blank(),  axis.text.y = theme_blank(),
                  axis.title.y = theme_blank(), axis.ticks = theme_blank(),
                  title = plot.titles[i])
           )

grid.newpage()
pushViewport(viewport(layout = grid.layout(nrow = 1, ncol = 4)))
## We can ignore the warnings from displaying the plots for now
for (i in 1:4)
    print(plots[[i]], vp = viewport(layout.pos.col = i))

However, in this case it is much more illuminating to look at the overall distribution:

ggplot(mccase, aes(duration)) +
    geom_histogram(binwidth = 0.05) +
    xlim(0, 3) +
    opts(title = "Duration of motorcycle policies") +
    annotate("text", 3.0, 1e4, label = "Histogram bin width = 0.05",
             size = 3, hjust = 1, vjust = 1)

![Duration of motorcycle policies](/files/PricingGLM/PricingGLM-2-6.png

Big spikes near 1/2 and 1 year (and in general spikes around 1/2 year multiples) suggests that either (1) this is not a random sample of policies or (2) there is a strong seasonal effect in the time of year people initially sign up for the policy.

Problem 3: Determine the relativities for claim frequency and severity

Determine the relativities for claim frequency and severity separately, by using GLMs; use the results to get relativities for the pure premium.

We first load the data (if needed) and create a data frame to hold the output, following the structure of Table 2.8 in the book.

library("data.table")
if (!exists("mccase.current"))
    load("mccase.current.RData")

case.2.4 <-
    data.frame(rating.factor =
               c(rep("Zone",        nlevels(mccase.current$rating.1)),
                 rep("MC class",    nlevels(mccase.current$rating.2)),
                 rep("Vehicle age", nlevels(mccase.current$rating.3)),
                 rep("Bonus class", nlevels(mccase.current$rating.4))),
               class =
               with(mccase.current,
                    c(levels(rating.1), levels(rating.2),
                      levels(rating.3), levels(rating.4))),
               ## These are the values from Table 2.8 in the book:
               relativity =
               c(7.678, 4.227, 1.336, 1.000, 1.734, 1.402, 1.402,
                 0.625, 0.769, 1.000, 1.406, 1.875, 4.062, 6.873,
                 2.000, 1.200, 1.000,
                 1.250, 1.125, 1.000),
               stringsAsFactors = FALSE)
print(case.2.4, digits = 3)

Close enough to the book.

First we set the contrasts so the baseline for the models is the level with the highest duration. This is the same approach we used before.

library("foreach")
new.cols <-
    foreach (rating.factor = paste("rating", 1:4, sep = "."),
             .combine = rbind) %do%
{
    totals <- mccase.current[, list(D = sum(duration),
                                    N = sum(antskad),
                                    C = sum(skadkost)),
                             by = rating.factor]
    n.levels <- nlevels(mccase.current[[rating.factor]])
    contrasts(mccase.current[[rating.factor]]) <-
        contr.treatment(n.levels)[rank(-totals[["D"]], ties.method = "first"), ]
    data.frame(duration = totals[["D"]],
               n.claims = totals[["N"]],
               skadkost = totals[["C"]])
}
case.2.4 <- cbind(case.2.4, new.cols)
rm(new.cols)

We next fit the models and combine the results as in Example 2.5 above. Here we also make a note of a simple measure of the goodness of fit: the residual deviance and the degrees of freedom. The frequency model is a good fit, the severity model is not.

## Model the frequency

model.frequency <-
    glm(antskad ~
        rating.1 + rating.2 + rating.3 + rating.4 + offset(log(duration)),
        data = mccase.current[duration > 0], family = poisson)
## Res. dev. 360 on 389 dof

rels <- coef( model.frequency )
rels <- exp( rels[1] + rels[-1] ) / exp( rels[1] )
case.2.4$rels.frequency <-
    c(c(1, rels[1:6])[rank(-case.2.4$duration[1:7], ties.method = "first")],
      c(1, rels[7:12])[rank(-case.2.4$duration[8:14], ties.method = "first")],
      c(1, rels[13:14])[rank(-case.2.4$duration[15:17], ties.method = "first")],
      c(1, rels[15:16])[rank(-case.2.4$duration[18:20], ties.method = "first")])

## Model the severity. We stick with the non-canonical link function
## for the time being.

model.severity <-
    glm(skadkost ~ rating.1 + rating.2 + rating.3 + rating.4,
        data = mccase.current[skadkost > 0,],
        family = Gamma("log"), weights = antskad)
## Res.dev. 516 on 164 dof

rels <- coef( model.severity )
rels <- exp( rels[1] + rels[-1] ) / exp( rels[1] )
case.2.4$rels.severity <-
    c(c(1, rels[1:6])[rank(-case.2.4$duration[1:7], ties.method = "first")],
      c(1, rels[7:12])[rank(-case.2.4$duration[8:14], ties.method = "first")],
      c(1, rels[13:14])[rank(-case.2.4$duration[15:17], ties.method = "first")],
      c(1, rels[15:16])[rank(-case.2.4$duration[18:20], ties.method = "first")])

## Combine the frequency and severity
case.2.4$rels.pure.prem <- with(case.2.4, rels.frequency * rels.severity)

Finally we convert, save, and compare with the current values.

## Convert to data.table
library("data.table")
case.2.4 <- data.table(case.2.4)

## Save
save(case.2.4, file = "case.2.4.RData")

## Compare with current values
print(case.2.4[,
               list(rating.factor, class, duration, n.claims,
                    skadkostK = round(skadkost / 1e3),
                    relativity, rels.pure.prem)],
      digits = 3)

It is really rather different from the current relativity, as we will also see below. But remember that the severity model was a poor fit: that would be my first place to start looking if staying within the insurance pricing framework outlined in this book.

Problem 4: Discussions

Just note here that the ratio for the relativity we calculated and for the existing one can be found as

with(case.2.4, max(rels.pure.prem) / min(rels.pure.prem)) # 3552
with(case.2.4, max(relativity) / min(relativity))         #   12

Note the huge range of relativities in our new model (3552 versus 12).

Insurance pricing is backwards and primitive, harking back to an era before (powerful) computers. One standard (and good) textbook on the topic is Non-Life Insurance Pricing with Generalized Linear Models by Esbjorn Ohlsson and Born Johansson (Amazon paid links UK | US). We have been doing some work in this area recently. Needing a robust internal training course and documented methodology, we have been working our way through the book again and converting the examples and exercises to R, the statistical computing and analysis platform. This is part of a series of posts containing elements of the R code.

Let’s get the preliminaries out of the way:

#!/usr/bin/Rscript
## PricingGLM-1.r - Code for Chapter 1 of "Non-Life Insurance Pricing with GLM"
## Copyright © 2012 CYBAEA Limited (http://www.cybaea.net/)

## @book{ohlsson2010non,
##   title={Non-Life Insurance Pricing with Generalized Linear Models},
##   author={Ohlsson, E. and Johansson, B.},
##   isbn={9783642107900},
##   series={Eaa Series: Textbook},
##   url={http://books.google.com/books?id=l4rjeflJ\_bIC},
##   year={2010},
##   publisher={Springer Verlag}
## }

Now we can get started.

Example 1.2

We grab the data for Table 1.2 from the book’s web site and store it as an R object with lots of good meta information.

################
### Example 1.2
con <- url("http://www2.math.su.se/~esbj/GLMbook/moppe.sas")
data <- readLines(con, n = 200L, warn = FALSE, encoding = "unknown")
close(con)
## Find the data range
data.start <- grep("^cards;", data) + 1L
data.end   <- grep("^;", data[data.start:999L]) + data.start - 2L
table.1.2  <- read.table(text = data[data.start:data.end],
                         header = FALSE, sep = "", quote = "",
                         col.names = c("premiekl", "moptva", "zon", "dur",
                             "medskad", "antskad", "riskpre", "helpre", "cell"),
                         na.strings = NULL,
                         colClasses = c(rep("factor", 3), "numeric",
                             rep("integer", 4), "NULL"),
                         comment.char = "")
rm(con, data, data.start, data.end)     # Cleanup
comment(table.1.2) <-
    c("Title: Partial casco moped insurance from Wasa insurance, 1994--1999",
      "Source: http://www2.math.su.se/~esbj/GLMbook/moppe.sas",
      "Copyright: http://www2.math.su.se/~esbj/GLMbook/")
## See the SAS code for this derived field
table.1.2$skadfre = with(table.1.2, antskad / dur)
## English language column names as comments:
comment(table.1.2$premiekl) <-
    c("Name: Class",
      "Code: 1=Weight over 60kg and more than 2 gears",
      "Code: 2=Other")
comment(table.1.2$moptva)   <-
    c("Name: Age",
      "Code: 1=At most 1 year",
      "Code: 2=2 years or more")
comment(table.1.2$zon)      <-
    c("Name: Zone",
      "Code: 1=Central and semi-central parts of Sweden's three largest cities",
      "Code: 2=suburbs and middle-sized towns",
      "Code: 3=Lesser towns, except those in 5 or 7",
      "Code: 4=Small towns and countryside, except 5--7",
      "Code: 5=Northern towns",
      "Code: 6=Northern countryside",
      "Code: 7=Gotland (Sweden's largest island)")
comment(table.1.2$dur)      <-
    c("Name: Duration",
      "Unit: year")
comment(table.1.2$medskad)  <-
    c("Name: Claim severity",
      "Unit: SEK")
comment(table.1.2$antskad)  <- "Name: No. claims"
comment(table.1.2$riskpre)  <-
    c("Name: Pure premium",
      "Unit: SEK")
comment(table.1.2$helpre)   <-
    c("Name: Actual premium",
      "Note: The premium for one year according to the tariff in force 1999",
      "Unit: SEK")
comment(table.1.2$skadfre)  <-
    c("Name: Claim frequency",
      "Unit: /year")
## Save results for later
save(table.1.2, file = "table.1.2.RData")
## Print the table (not as pretty as the book)
print(table.1.2)
################

That was easy. Now for something a little harder.

Example 1.3

Here we are concerned with replicating Table 1.4. We do it slowly, step-by-step, for pedagogical reasons.

################
### Example 1.3
if (!exists("table.1.2"))
    load("table.1.2.RData")
## We calculate each of the columns individually and slowly here
## to show each step

## First we have simply the labels of the table
rating.factor <-
    with(table.1.2,
         c(rep("Vehicle class", nlevels(premiekl)),
           rep("Vehicle age", nlevels(moptva)),
           rep("Zone", nlevels(zon))))

## The Class column
class.num <- with(table.1.2, c(levels(premiekl), levels(moptva), levels(zon)))

## The Duration is the sum of durations within each class
duration.total <-
    c(with(table.1.2, tapply(dur, premiekl, sum)),
      with(table.1.2, tapply(dur, moptva, sum)),
      with(table.1.2, tapply(dur, zon, sum)))

## Calculate relativities in the tariff
## The denominator of the fraction is the class with the highest exposure
## (i.e. the maximum total duration): we make that explicit with the
## which.max() construct.  We also set the contrasts to use this as the base,
## which will be useful for the glm() model later.
class.base <- which.max(duration.total[1:2])
age.base   <- which.max(duration.total[3:4])
zone.base  <- which.max(duration.total[5:11])

rt.class <- with(table.1.2, tapply(helpre, premiekl, sum))
rt.class <- rt.class / rt.class[class.base]
rt.age   <- with(table.1.2, tapply(helpre, moptva, sum))
rt.age   <- rt.age / rt.age[age.base]
rt.zone  <- with(table.1.2, tapply(helpre, zon, sum))
rt.zone  <- rt.zone / rt.zone[zone.base]

contrasts(table.1.2$premiekl) <-
    contr.treatment(nlevels(table.1.2$premiekl))[rank(-duration.total[1:2],
                                                      ties.method = "first"), ]
contrasts(table.1.2$moptva) <-
    contr.treatment(nlevels(table.1.2$moptva))[rank(-duration.total[3:4],
                                                    ties.method = "first"), ]
contrasts(table.1.2$zon) <-
    contr.treatment(nlevels(table.1.2$zon))[rank(-duration.total[5:11],
                                                 ties.method = "first"), ]

The contrasts could also have been set with the base= argument, e.g. contrasts(table.1.2$zon) <- contr.treatment(nlevels(table.1.2$zon), base = zone.base), which would be closer in spirit to the SAS code. But I like the idiom presented here where we follow the duration order; it also extends well to other (i.e. not treatment) contrasts. I just wish rank() had a decreasing= argument like order() which I think would be clearer than using rank(-x) to get a decreasing sort order.

That was the easy part. At this stage in the book you are not really expected to understand the next step so do not despair! We just show how easy it is to replicate the SAS code in R. An alternative approach using direct optimization is outlined in Exercise 1.3 below.

## Relativities of MMT; we use the glm approach here as per the book’s
## SAS code at http://www2.math.su.se/~esbj/GLMbook/moppe.sas
m <- glm(riskpre ~ premiekl + moptva + zon, data = table.1.2,
         family = poisson("log"), weights = dur)

## If the next line is a mystery then you need to
## (1) read up on contrasts or
## (2) remember that the link function is log() which is why we use exp here
rels <- exp( coef(m)[1] + coef(m)[-1] ) / exp(coef(m)[1])

rm.class <- c(1, rels[1])               # See rm.zone below for the
rm.age   <- c(rels[2], 1)               # general approach
rm.zone  <- c(1, rels[3:8])[rank(-duration.total[5:11], ties.method = "first")]

## Create and save the data frame
table.1.4 <-
    data.frame(Rating.factor = rating.factor, Class = class.num,
               Duration = duration.total,
               Rel.tariff = c(rt.class, rt.age, rt.zone),
               Rel.MMT    = c(rm.class, rm.age, rm.zone))
save(table.1.4, file = "table.1.4.RData")
print(table.1.4, digits = 3)
rm(rating.factor, class.num, duration.total, class.base, age.base, zone.base,
   rt.class, rt.age, rt.zone, rm.class, rm.age, rm.zone, m, rels)
################

The result is something like this:

Note the rather unusual and apparently inconsistent rounding in the book: 147, 1.66, and 5.16 would be better as 148 (the value is 147.5), 1.67, and 5.17.

Exercise 1.3

Here it gets interesting as we get a different value from the authors. Possibly a small bug on our part but at least we provide the code for you to check. So if you spot a problem let us know in the comments.

################
## Exercise 1.3

## The values from the book
g0  <- 0.03305
g12 <- 2.01231
g22 <- 0.74288
dim.names <- list(Milage = c("Low", "High"),
                  Age = c("New", "Old"))
pyears <- matrix(c(47039, 56455, 190513, 28612), nrow = 2,
                 dimnames = dim.names)
claims <- matrix(c(0.033, 0.067, 0.025, 0.049), nrow = 2,
                 dimnames = dim.names)

## Function to calculate the error of the estimate
GvalsError <- function (gvals) {
    ## The current estimates
    g0  <- gvals[1]
    g12 <- gvals[2]
    g22 <- gvals[3]
    ## The current estimates in convenient matrix form
    G  <- matrix(c(1, 1, g12, g22), nrow = 2)
    G1 <- matrix(c(1, g12), nrow = 2, ncol = 2)
    G2 <- matrix(c(1, g22), nrow = 2, ncol = 2, byrow = TRUE)
    ## The calculated values
    G0  <- addmargins(claims * pyears)["Sum", "Sum"] / ( sum(pyears * G1 * G2) )
    G12 <- addmargins(claims * pyears)["High", "Sum"] /
        ( g0 * addmargins(pyears * G2)["High", "Sum"] )
    G22 <- addmargins(claims * pyears)["Sum", "Old"] /
        ( g0 * addmargins(pyears * G1)["Sum", "Old"] )
    ## The sum of squared errors
    error <- (g0 - G0)^2 + (g12 - G12)^2 + (g22 - G22)^2
    return(error)
}

## Minimize the error function to obtain our estimate
gamma <- optim(c(g0, g12, g22), GvalsError)
stopifnot(gamma$convergence == 0)
gamma <- gamma$par

values <- data.frame(legend = c("Our calculation", "Book value"),
                     g0  = c(gamma[1], g0),
                     g12 = c(gamma[2], g12),
                     g22 = c(gamma[3], g22),
                     row.names = "legend")
print(values, digits = 4)

## Close, but not the same.

rm(g0, g12, g22, dim.names, pyears, claims, gamma, values)
################

The resulting table is something like:

Close, but not the same. Perhaps they used a different error function.

The benefits of R is productivity. We want to spend time on the actions from the analytical insights, not the coding, and we choose our tool accordingly. Being a consulting type organization it is easier for us to attract and retain talent.

The advantages of SAS/SPSS in a commercial environment

1. You can buy the tool for money

Big corporations have procurement departments who do not have a process for free software. Also software spend goes on the balance sheet in a way that the CFO prefers to people but something like R will take a little talent to set up initally. (And yes, we know the Revolutions guys well, but they are not really credible in Europe yet.)

This will change as (a) companies become more mature in their procurement and as (b) commercial support for R improves. (On the latter point, Oracle’s R integration to the database is great news.)

2. You can recruit for the commercial tool

1. Recruiters are familiar with SAS and SPSS but not with R so it is easier to brief them and to get good quality CVs. This will change and R becomes ever more popular and prevalent. [And yes, we could in theory change recruiters to someone clued in, but again in large corporations there are procurement processes to be followed and existing agreements to be honoured so it will all take months or years.]

2. There are recognised training programmes for SPSS and (especially) SAS which makes it easier to recruit the technical skills. How do you know what somebody knows when they say they “know R”? How do you even begin to quantify it from a CV? How do you separate the guy who downloaded the tool and just read An Introduction to R from the Frank Harrells of this world?

   Yes, I would argue (and in fact have argued in Commercial Analytics: The Capabilities) that technical skill is not the most important in an analyst (and can be learned anyhow) but it does help filter the CVs and, you guessed it, fits well with the corporation’s processes.

   (One reason we use R internally is that we find that it is, on average, a more interesting type of analyst who is proficient in that tool. It seems to encourage curiosity and love or learning in a way that menu-based tools do not.)

   I think the commercial R companies are really missing a trick here to provide recognised certification.

3. You can’t search for R. Seriously: try searching for R on LinkedIn. Much easier to find SAS / SPSS skills in a large CV database (like LinkedIn where this discussion started).

3. You can recruit for the commercial tool.

Yes I know I already said that but there is another reason why this is critical.

R takes talent to use. (That is kind of why we like it.) It takes talent to maintain.

My problem as the manager of a commercial analytical insights team is that it is very hard for me to retain that talent. Think about it: what can I offer in terms of career progression? If you are an analyst you might become a senior analyst but you will always be an analyst. There are no examples of a way up the organization (except perhaps out through IT and then up to CIO). [This too will change with time.] And new challenges: yes, some, but we are not a research university and it tends to be the same few problem types that we are always working on. So if you are an analyst looking for new challenges and more pay, the best thing—the logical and rational thing to do—is to get a new job. And your time with Big Corporation will look good on your CV and you will probably land the job easily.

We can help

If you want to set up a commercial analytical group we can help you get it right first time. The right people, the right processes, the right infrastructure and most importantly the right results. We have done it before and are not tied to any specific tool or vendor.

If you want to improve or enhance your existing analytical teams, then we can Reboot your Analytics to deliver both rapid and sustained commercial results.

And if you just want the results we can provide commercial analytics as a service where we provide the insights and then work with you to turn those insights into commercial actions and better understanding of your business, markets, and customers, leaving you to focus on what you do best.

Contact us now and get results from your analytics.

How to lose your data with save()

I am using Windows on my travel laptop and Linux on my workstation. To speed things up on the latter and make use of my many (well, four) cores, I use the ‘multicore’ package, which I do not have available on the Windows machine.

To illustrate the problem with the save file format, I created a file on the Linux machine simply as:

library("multicore")
a <- list(data = 1:10, fun = mclapply)
save(a, file = "a.RData")

What could be simpler? The mclapply is a function from the ‘{multicore}’ package but it clearly has no impact on the stored data. (We will show a more realistic example below – work with me here.)

But try to open the save file on a machine without the package installed, like my Windows laptop, and you get:

Error in loadNamespace(name) : there is no package called 'multicore'

There is no way of getting to your precious data without installing the missing package.

If the package has been withdrawn or is no longer available then your data is basically lost.

What can you do?

Some suggestions from the helpful people on R-help:

1. (Uwe Ligges): You could try to rewrite ./src/main/saveload.R and serialize.R to extract only the parts you need. “This is probably not worth the effort.”

2. (Prof. Brian Ripley): You could try installing the missing package; R CMD INSTALL --fake should be sufficient to let you load the data. Also suggests that the proposal above would be very hard indeed.

3. (Martin Morgan): Don’t store package functions with your code.

That is three good answers from three of the heavy-weights in the R community. Thank you all!

Martin’s comment is worth expanding. We can change the above example to:

library("multicore")
computeFunction <- function(...) {
    if (require(multicore)) mclapply(...)
    else lapply(...) 
}
a <- list(data = 1:10, fun = computeFunction)
save(a, file = "a.RData")

Now everything works fine! No data is horribly lost: the file loads fine on the {multicore}-less machine.

And for the more realistic example, I had been using caret::rfe as Martin knew in the example he provided:

library("caret")
data(BloodBrain)

x <- scale(bbbDescr[,-nearZeroVar(bbbDescr)])
x <- x[, -findCorrelation(cor(x), .8)]
x <- as.data.frame(x)

set.seed(1)
lmProfile <- rfe(x, logBBB,
                 sizes = c(2:25, 30, 35, 40, 45, 50, 55, 60, 65),
                 rfeControl = rfeControl(functions = lmFuncs,
                   number = 5,
                   computeFunction = mclapply))
save(lmProfile, file = "lmProfile.RData")

Slightly less obvious that there is a reference to the external namespace in this code, but easy enough to see if you know what to look for.

For old files I will use the R CMD INSTALL --fake suggestion, but for new data I am going with the last approach and using a computeFunction like this:

### MCCompute: A computeFunction for caret::rfeControl and caret::trainControl 
### that does not leave a reference to the multicore package in the save file
MCCompute <- function(X, FUN, ...) {
    FUN <- match.fun(FUN)
    if (!is.vector(X) || is.object(X)) 
        X <- as.list(X)
    if (require("multicore")) mclapply(X, FUN, ...)
    else lapply(X, FUN, ...)
}

I know that Max Kuhn is rewriting the caret package which should make this a moot point in the near future for that specific case. But the indirection approach is generally useful and will also be relevant in other situations.

Recommendations

My recommendations:

1. Save data in a data format, not using the save() function which is really for objects (data and code). Suitable formats include CSV and variants, HDF5, and CDF, as well as others.

2. Avoid references to packages in your objects by using the one level indirection trick exemplified by the MCCompute function shown.

What is your approach? Suggestions in the comments below, please.

Spreadsheets are good for some things, but analysing data is not one of them. The example data in the table on the right is from Jeffrey S. Simonoff, “Statistical analysis using Microsoft Excel” (2008), and looks at first (and maybe even second) glance like a reasonable set of observations.

However, the predictors are (accidentally) collinear so no meaningful fit is possible, unless one of them are dropped. We see that very easily if we try to do the analysis using the R statistical computing and analysis platform:

> d <- read.delim("clipboard")  # Read DATA range from clipboard
> summary(lm(Y ~ ., data = d))

Call:
lm(formula = Y ~ ., data = d)

Residuals:
    Min      1Q  Median      3Q     Max 
-11.222  -5.821  -2.546   3.171  40.750 

Coefficients: (1 not defined because of singularities)
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   4.1945     3.9749   1.055    0.296
X1            0.3862     0.5652   0.683    0.497
X2            0.2308     3.1590   0.073    0.942
X3            3.7072     2.9922   1.239    0.221
X4                NA         NA      NA       NA

Residual standard error: 10.14 on 50 degrees of freedom
Multiple R-squared: 0.04767,    Adjusted R-squared: -0.009466 
F-statistic: 0.8343 on 3 and 50 DF,  p-value: 0.4814

Everybody likes to pick on Excel, so let us load the data into version 3.3.2 of LibreOffice, the free Open Source personal productivity suite, instead. It faithfully implements many of the worst features of Excel. You can grab a copy of the spreadsheet GS-spreadsheet-error.ods yourself and see the results. The relevant function in both Excel and LibreOffice for linear regression is LINEST and applying it to the data set give us:

Of the 16 values returned by the function, fully 12 of them are incorrect (highlighted in red), and the ‘#VALUE!’ entries are the only thing that suggests we may have a problem. (The ‘#N/A’ values are a feature of the function and not a problem.) Excluding the X4 values from the function call gives meaningful (and correct) results:

There is so much wrong with doing even this trivial analysis in a spreadsheet that it is hard to know where to start. Some of the problems:

Garbage results instead of errors

Instead of giving meaningful errors or warnings, the spreadsheets simply produce garbage results. This is nearly impossible to debug.

No help on how to correct the problem

In the erroneous results of the first figure, there is no clue, no hint, no help to figure out how to correct the problem. You could argue about R correcting the issue ”automagically”, but at least it finds a solution to the problem and tells you about it.

Error prone output formats

I put in the row and column headings because otherwise it is just too hard to read the data. Where does the function stuff the F statistics again?

And don’t get me started on version control and documentation. Don’t even mention that the maths in Excel are wrong. Don’t bring up the list of academically researched spreadsheet horror stories. Just remember: Friends do not let friends do data analysis in spreadsheets.

We do not have the full set of data yet, so this is a simple warm-up session to predict the days in hospital in year 2 based on the year 1 data.

Prerequisites

Obviously you need to have R installed, and you should also have signed up for the competition (be sure to read the terms carefully) and downloaded and extracted the release 1 data file.

Data preparation

Let’s load the data into R and do some basic housekeeping:

#!/usr/bin/Rscript
## example001.R - simple benchmarks for the HHP
## Copyright © 2011 CYBAEA Limited - http://www.cybaea.net/

##############################
#### DATA PREPARATION

##++++
## Members
members   <- read.csv(file = "HHP_release1/Members_Y1.csv",
                      colClasses = rep("factor", 3),
                      comment.char = "")
##----
##++++
## Claims
claims.Y1 <- read.csv(file = "HHP_release1/Claims_Y1.csv",
                      colClasses = c(
                          rep("factor", 7),
                          "integer",    # paydelay
                          "character",  # LengthOfStay
                          "character",  # dsfs
                          "factor",     # PrimaryConditionGroup
                          "character"   # CharlsonIndex
                          ),
                      comment.char = "")
## Utility function
make.numeric <- function(cv, FUN = mean) {
### make a character vector numeric by splitting on '-'
    sapply(strsplit(gsub("[^[:digit:]]+",
                         " ",
                         cv,
                         perl = TRUE),
                    " ",
                    fixed = TRUE),
           function(x) FUN(as.numeric(x)))
}
## Length of stay as days
{
    z <- make.numeric(claims.Y1$LengthOfStay)
    z.week <- grepl("week", claims.Y1$LengthOfStay, fixed = TRUE)
    z[z.week] <- z[z.week] * 7          # Weeks are 7 days
    z[is.nan(z)] <- 0
    claims.Y1$LengthOfStay.days <- z
}
los.levels <- c("", "1 day", sprintf("%d days", 2:6),
                "1- 2 weeks", "2- 4 weeks", "4- 8 weeks", "8-12 weeks",
                "12-26 weeks", "26+ weeks")
stopifnot(all(claims.Y1$LengthOfStay %in% los.levels))
claims.Y1$LengthOfStay <- factor(claims.Y1$LengthOfStay,
                                 levels = los.levels,
                                 labels = c("0 days", los.levels[-1]),
                                 ordered = TRUE)
## Months since first claim
claims.Y1$dsfs.months <- make.numeric(claims.Y1$dsfs)
## dsfs is an ordered factor and gives the ordering of the claims
dsfs.levels <- c("0- 1 month", sprintf("%d-%2d months", 1:11, 2:12))
claims.Y1$dsfs <- factor(claims.Y1$dsfs, levels = dsfs.levels, ordered = TRUE)
## Index as numeric
claims.Y1$CharlsonIndex.numeric <- make.numeric(claims.Y1$CharlsonIndex)
claims.Y1$CharlsonIndex <- factor(claims.Y1$CharlsonIndex, ordered = TRUE)
##----
##++++
## Days in hospital
dih.Y2    <- read.csv(file = "HHP_release1/DayInHospital_Y2.csv",
                      colClasses = c("factor", "integer"),
                      comment.char = "")
names(dih.Y2)[1] <- "MemberID"          # Fix broken file
##----
save(members, claims.Y1, dih.Y2,
     file = "HHPR1.RData")
##############################

Scoring

We will need a function to score our predictions p against the actual values a. The formula is on the evaluation page and we implement it as:

#!/usr/bin/Rscript
## example001.R - simple benchmarks for the HHP
## Copyright © 2011 CYBAEA Limited - http://www.cybaea.net/

##############################
#### FUNCTION TO CALCULATE SCORE
HPPScore <- function(p, a) {
### Scorng function after
### http://www.heritagehealthprize.com/c/hhp/Details/Evaluation
### Base 10 log from http://www.heritagehealthprize.com/forums/default.aspx?g=posts&m=2226#post2226
    sqrt(mean((log(1 + p, 10) - log(1 + a, 10))^2))
}
##############################

The simplest benchmarks

The simplest models don’t really model at all: they just use the average and are simple benchmarks.

#!/usr/bin/Rscript
## example001.R - simple benchmarks for the HHP
## Copyright © 2011 CYBAEA Limited - http://www.cybaea.net/

y <- dih.Y2$DaysInHospital_Y2           # Actual
p <- rep(mean(y), NROW(dih.Y2))
cat(sprintf("Score using mean  : %8.6f\n", HPPScore(p, y)))
# Score using mean  : 0.278725

p <- rep(median(y), NROW(dih.Y2))
cat(sprintf("Score using median: %8.6f\n", HPPScore(p, y)))
# Score using median: 0.267969

Simple single-variable linear models

OK, a model that doesn’t use past data isn’t much of a model, so let’s improve on that:

#!/usr/bin/Rscript
## example001.R - simple benchmarks for the HHP
## Copyright © 2011 CYBAEA Limited - http://www.cybaea.net/
library("reshape2")

vars <- dcast(claims.Y1, MemberID ~ ., sum, value_var = "LengthOfStay.days")
names(vars)[2] <- "LengthOfStay"
data <- merge(vars, dih.Y2)

model <- lm(DaysInHospital_Y2 ~ LengthOfStay, data = data)
p <- predict(model)
cat(sprintf("Score using lm(LengthOfStay): %8.6f\n", HPPScore(p, y)))
# Score using lm(LengthOfStay): 0.279062

model <- glm(DaysInHospital_Y2 ~ LengthOfStay,
             family = quasipoisson(),
             data = data)
p <- predict(model, type="response")
cat(sprintf("Score using glm(LengthOfStay): %8.6f\n", HPPScore(p, y)))
# Score using glm(LengthOfStay): 0.278914

Let the competition begin.

Having looked at feature selection using the Boruta package and feature selection using the caret package separately, we now consider the performance of the two approaches.

For our tests we will use an artificially constructed trivial data sets that the automated process should have no problems with (but we will be disappointed later on this expectation, as we will see). The data set has an equal number of normal and uniform random variables with mean 0 and variance 1 of which 20% are used for the target variable. There are 10 time as many observations as variables. We create a function to set this up:

#| label: "fig-data"
make.data <- function(n.var, m.rand = 5, m.obs = 10) {
  n.col <- n.var * m.rand
  n.obs <- n.col * m.obs * 2
  x <- data.frame(
    N = matrix(rnorm(n = n.col * n.obs),
      nrow = n.obs, ncol = n.col
    ),
    U = matrix(runif(
      n = n.col * n.obs,
      min = -sqrt(3), max = sqrt(3)
    ), n.obs, n.col)
  )
  deps.n <- 1:n.var
  deps.u <- (1 + n.col):(n.var + n.col)
  y <- rowSums(as.matrix(x[, c(deps.n, deps.u)]))
  x <- cbind(x, Y = factor(y >= 0, labels = c("N", "P")))
  attr(x, "vars") <- names(x)[c(deps.n, deps.u)]
  return(x)
}

The Boruta package

Then we run a test using the Boruta package for different sizes:

#!/usr/bin/Rscript
## bench.R - benchmark Boruta package
## Copyright © 2010 Allan Engelhardt (https://www.cybaea.net/)
run.name <- "bench-1"
library("Boruta")

set.seed(1)

sizes <- c(1:10, 10 * (2:10), 100 * (2:10), 1e3 * (2:10))
n.sizes <- length(sizes)
bench <- data.frame(n.vars = sizes, elapsed = NA, right = NA, wrong = NA)
file.name <- paste(run.name, "RData", sep = ".")

for (n in 1:length(sizes)) {
  size <- sizes[n]
  cat(sprintf("[%s] Size = %3d: ", as.character(Sys.time()), size))
  tries <- max(3, round(10 / size, 0))
  n.right <- 0
  n.wrong <- 0
  elapsed <- 0
  for (try in 1:tries) {
    cat(tries - try, ".", sep = "")
    x <- make.data(size)
    x.vars <- attr(x, "vars")
    elapsed <- elapsed +
      system.time({
        b <- Boruta(x[, -NCOL(x)], x[, NCOL(x)])
      })["elapsed"]
    b.vars <- names(b$finalDecision)[b$finalDecision != "Rejected"]
    n.right <- n.right + length(intersect(b.vars, x.vars))
    n.wrong <- n.wrong + length(setdiff(b.vars, x.vars))
  }
  elapsed <- elapsed / tries
  cat(" Elapsed = ", round(elapsed, 0), " seconds\n", sep = "")
  n.right <- n.right / tries
  n.wrong <- n.wrong / tries
  bench[n, ] <- c(size, elapsed, n.right, n.wrong)
  save(bench, file = file.name, ascii = FALSE, compress = FALSE)
}

print(bench)

As it turned out, our expectations for the size of data set we could handle were wildly optimistic and we killed the process at size 30. We add to the data set a field with the total number of variables in the x data set and plot the results.

load(file = "bench-1.RData")
bench <- na.omit(bench)
bench$n.elem <- bench$n.var^2 * 1e3
plot(elapsed ~ n.elem,
  data = bench, type = "b",
  main = "Feature selections with Boruta",
  sub = "Elapsed time versus number of data elements",
  log = "xy",
  xlab = "Elements in data set", ylab = "Elapsed time (seconds)"
)

A quick check using summary(lm(log(elapsed) ~ log(n.elem), data = bench)) shows us a linear scaling with the number of elements (slope is 1.01 with standard error 0.025 and adjusted R² 0.993). The algorithm selects all the right features up to n.vars = 10 when it starts to miss some of them:

A higher accuracy in the feature selection for the larger problems could presumably be achieved by adjusting the maxRuns and perhaps confidence parameters on the Boruta call.

In summary, the Boruta package performs well up to about 20 features out of 100 (n.vars = 10) which runs in about 11 minutes on my machine. If we changed the technical implementation to support multicore, MPI, and other parallel frameworks, then the out of the box settings would be useful up to n.vars of 20 or 30 (40-60 features out of 200-300) which an 8-core machine should be able to complete in 20 minutes or so.

This is still a lot less than the size of data sets we normally work with. (Our usual benchmark is 15,000 variables and 50,000 observations.)

The caret package

One of the nice features of the caret package is that is supports most parallel processing frameworks out of the box, but for comparison with the previous analysis we will (somewhat unfairly) not use that here. The setup is then quite simple, using the same make.data function as before.

#!/usr/bin/Rscript
## bench.R - benchmark caret package
## Copyright © 2010 Allan Engelhardt (http://www.cybaea.net/)
run.name <- "bench-2"
library("caret")
library("randomForest")
set.seed(1)

control <- rfeControl(
  functions = rfFuncs, verbose = FALSE,
  returnResamp = "final"
)

## if ( require("multicore", quietly = TRUE, warn.conflicts = FALSE) ) {
##     control$workers <- multicore:::detectCores()
##     control$computeFunction <- mclapply
##     control$computeArgs <- list(mc.preschedule = FALSE, mc.set.seed = FALSE)
## }

our.sizes <- c(2:10, 10 * (2:10), 100 * (2:10), 1e3 * (2:10))
n.sizes <- length(our.sizes)
bench <- data.frame(n.vars = our.sizes, elapsed = NA, right = NA, wrong = NA)
file.name <- paste(run.name, "RData", sep = ".")

for (n in 1:length(our.sizes)) {
  size <- our.sizes[n]
  cat(sprintf("[%s] Size = %3d: ", as.character(Sys.time()), size))
  tries <- max(3, round(10 / size, 0))
  n.right <- 0
  n.wrong <- 0
  elapsed <- 0
  for (try in 1:tries) {
    cat(tries - try, ".", sep = "")
    x <- make.data(size)
    x.vars <- attr(x, "vars")
    elapsed <- elapsed +
      system.time({
        p <- rfe(x[, -NCOL(x)], x[, NCOL(x)],
          sizes = 1:(2 * size), rfeControl = control
        )
      })["elapsed"]
    p.vars <- predictors(p)
    n.right <- n.right + length(intersect(p.vars, x.vars))
    n.wrong <- n.wrong + length(setdiff(p.vars, x.vars))
  }
  elapsed <- elapsed / tries
  cat(" Elapsed = ", round(elapsed, 0), " seconds\n", sep = "")
  n.right <- n.right / tries
  n.wrong <- n.wrong / tries
  bench[n, ] <- c(size, elapsed, n.right, n.wrong)
  save(bench, file = file.name, ascii = FALSE, compress = FALSE)
}

print(bench)

This uses the randomForest classifier from the package of the same name. To use the ipredbagg bagging classifier from Andrea Peters and Torsten Hothorn’s ipred: Improved Predictors package we simply change the control object to:

control <- rfeControl(
  functions = treebagFuncs, verbose = FALSE,
  returnResamp = "final"
)

As usual, we were widely optimistic in our guesses for the size of problems we could handle, and had to abort the run.

Remember that the right number of significant features are 2 * n.vars and we see that the caret package apparently always miss one feature in its selection, which is very odd and possibly a bug. It is less likely to select the wrong features than Boruta, but that could be partially due to “Tentative” data in Boruta. Timing-wise, performance is a little worse in the non-parallel situation but realistically of course a lot better than Boruta depending on the number of cores on your processor or nodes in your cluster.

Neither approach is suitable out of the box for the sizes of data sets that we normally work with.

The caret package provides a very flexible framework for the analysis as we shall see, but first we set up the artificial test data set as in the previous article.

## Feature-bc.R - Compare Boruta and caret feature selection
## Copyright © 2010 Allan Engelhardt (http://www.cybaea.net/)
run.name <- "feature-bc"
library("caret")

## Load early to get the warnings out of the way:
library("randomForest")
library("ipred")
library("gbm")

set.seed(1)

## Set up artificial test data for our analysis
n.var <- 20
n.obs <- 200
x <- data.frame(V = matrix(rnorm(n.var * n.obs), n.obs, n.var))
n.dep <- floor(n.var / 5)
cat("Number of dependent variables is", n.dep, "\n")
m <- diag(n.dep:1)

## These are our four test targets
y.1 <- factor(ifelse(x$V.1 >= 0, "A", "B"))
y.2 <- ifelse(rowSums(as.matrix(x[, 1:n.dep]) %*% m) >= 0, "A", "B")
y.2 <- factor(y.2)
y.3 <- factor(rowSums(x[, 1:n.dep] >= 0))
y.4 <- factor(rowSums(x[, 1:n.dep] >= 0) %% 2)

The flexibility of the caret package is to a large extent implemented by using control objects. Here we specify to use the randomForest classification algorithm (which is also what Boruta uses) and if the multicore package is available then we use that for extra perfomance (you can also use MPI etc – see the documentation):

control <- rfeControl(
  functions = rfFuncs, method = "boot", verbose = FALSE,
  returnResamp = "final", number = 50
)

if (require("multicore", quietly = TRUE, warn.conflicts = FALSE)) {
  control$workers <- multicore:::detectCores()
  control$computeFunction <- mclapply
  control$computeArgs <- list(mc.preschedule = FALSE, mc.set.seed = FALSE)
}

We will consider from one to six features (using the sizes variable) and then we simply let it lose:

sizes <- 1:6

## Use randomForest for prediction
profile.1 <- rfe(x, y.1, sizes = sizes, rfeControl = control)
cat("rf     : Profile 1 predictors:", predictors(profile.1), fill = TRUE)
profile.2 <- rfe(x, y.2, sizes = sizes, rfeControl = control)
cat("rf     : Profile 2 predictors:", predictors(profile.2), fill = TRUE)
profile.3 <- rfe(x, y.3, sizes = sizes, rfeControl = control)
cat("rf     : Profile 3 predictors:", predictors(profile.3), fill = TRUE)
profile.4 <- rfe(x, y.4, sizes = sizes, rfeControl = control)
cat("rf     : Profile 4 predictors:", predictors(profile.4), fill = TRUE)

The results are:

rf     : Profile 1 predictors: V.1 V.16 V.6
rf     : Profile 2 predictors: V.1 V.2
rf     : Profile 3 predictors: V.4 V.1 V.2
rf     : Profile 4 predictors: V.10 V.11 V.7

If you recall the feature selection with Boruta article, then the results there were

Profile 1: V.1, (V.16, V.17)
Profile 2: V.1, V.2, V,3, (V.8, V.9, V.4)
Profile 3: V.1, V.4, V.3, V.2, (V.7, V.6)
Profile 4: V.10, (V.11, V.13)

To show the flexibility of caret, we can run the analysis with another of the built-in classifiers:

## Use ipred::ipredbag for prediction
control$functions <- treebagFuncs
profile.1 <- rfe(x, y.1, sizes = sizes, rfeControl = control)
cat("treebag: Profile 1 predictors:", predictors(profile.1), fill = TRUE)
profile.2 <- rfe(x, y.2, sizes = sizes, rfeControl = control)
cat("treebag: Profile 2 predictors:", predictors(profile.2), fill = TRUE)
profile.3 <- rfe(x, y.3, sizes = sizes, rfeControl = control)
cat("treebag: Profile 3 predictors:", predictors(profile.3), fill = TRUE)
profile.4 <- rfe(x, y.4, sizes = sizes, rfeControl = control)
cat("treebag: Profile 4 predictors:", predictors(profile.4), fill = TRUE)

This gives:

treebag: Profile 1 predictors: V.1 V.16
treebag: Profile 2 predictors: V.2 V.1
treebag: Profile 3 predictors: V.1 V.3 V.2
treebag: Profile 4 predictors: V.10 V.11 V.1 V.7 V.13

And of course, if you have your own favourite model class that is not already implemented, then you can easily do that yourself. We like gbm from the package of the same name, which is kind of silly to use here because it provides variable importance automatically as part of the fitting process, but may still be useful. It needs numeric predictors so we do:

## Use gbm for prediction
y.1 <- as.numeric(y.1) - 1
y.2 <- as.numeric(y.2) - 1
y.3 <- as.numeric(y.3) - 1
y.4 <- as.numeric(y.4) - 1

gbmFuncs <- treebagFuncs
gbmFuncs$fit <- function(x, y, first, last, ...) {
  library("gbm")
  n.levels <- length(unique(y))
  if (n.levels == 2) {
    distribution <- "bernoulli"
  } else {
    distribution <- "gaussian"
  }
  gbm.fit(x, y, distribution = distribution, ...)
}
gbmFuncs$pred <- function(object, x) {
  n.trees <- suppressWarnings(gbm.perf(object,
    plot.it = FALSE,
    method = "OOB"
  ))
  if (n.trees <= 0) n.trees <- object$n.trees
  predict(object, x, n.trees = n.trees, type = "link")
}
control$functions <- gbmFuncs

n.trees <- 1e2 # Default value for gbm is 100

profile.1 <- rfe(x, y.1,
  sizes = sizes, rfeControl = control, verbose = FALSE,
  n.trees = n.trees
)
cat("gbm    : Profile 1 predictors:", predictors(profile.1), fill = TRUE)
profile.2 <- rfe(x, y.2,
  sizes = sizes, rfeControl = control, verbose = FALSE,
  n.trees = n.trees
)
cat("gbm    : Profile 2 predictors:", predictors(profile.2), fill = TRUE)
profile.3 <- rfe(x, y.3,
  sizes = sizes, rfeControl = control, verbose = FALSE,
  n.trees = n.trees
)
cat("gbm    : Profile 3 predictors:", predictors(profile.3), fill = TRUE)
profile.4 <- rfe(x, y.4,
  sizes = sizes, rfeControl = control, verbose = FALSE,
  n.trees = n.trees
)
cat("gbm    : Profile 4 predictors:", predictors(profile.4), fill = TRUE)

And we get the results below:

gbm    : Profile 1 predictors: V.1 V.10 V.11 V.12 V.13
gbm    : Profile 2 predictors: V.1 V.2
gbm    : Profile 3 predictors: V.4 V.1 V.2 V.3 V.7
gbm    : Profile 4 predictors: V.11 V.10 V.1 V.6 V.7 V.18

It is all good and very flexible, for sure, but I can’t really say it is better than the Boruta approach for these simple examples.

@online{2010,
  author = {},
  title = {Feature Selection: {Using} the Caret Package},
  date = {2010-11-16},
  url = {https://www.cybaea.net/Journal/2010/11/16/Feature-selection-Using-the-caret-package.html},
  langid = {en-GB-oxendict}
}

In this article we take a first look at the problem of all-relevant feature selection using the Boruta package by Miron B. Kursa and Witold R. Rudnicki. This package is developed for the R statistical computing and analysis platform.

Background

All-relevant feature selection is extremely useful for commercial data miners. We deploy it when we want to understand the mechanisms behind the behaviour or subject of interest, rather than just building a black-box predictive model. This understanding leads us to a better appreciation of our customers (or other subject under investigation) and not just how, but why they behave as they do, which is useful for all areas of the business, including strategy and product development. More narrowly, it also help us define the variables that we want to observe which is what will really make a difference in our ability to predict behaviour (as opposed to, say, run the data mining application a little longer).

I really like the theoretical approach that the Boruta package tries to implement. It is based on the more general idea that by adding randomness to a system and then collecting results from random samples of the bigger system, one can actually reduce the misleading impact of randomness in the original sample.

For the implementation, the Boruta package relies on a random forest classification algorithm. This provides an intrinsic measure of the importance of each feature, known as the Z score. While this score is not directly a statistical measure of the significance of the feature, we can compare it to random permutations of (a selection of) the variables to test if it is higher than the scores from random variables. This is the essence of the implementation in Boruta.

The tests

This article is a first investigation into the performance of the Boruta package. For this initial examination we will use a test data sample that we can control so we know what is important and what is not. We will consider 200 observations of 20 normally distributed random variables:

run.name <- "feature-1"
library("Boruta")
set.seed(1)
## Set up artificial test data for our analysis
n.var <- 20
n.obs <- 200
x <- data.frame(V=matrix(rnorm(n.var*n.obs), n.obs, n.var))

Normal distribution has the advantage of simplicity, but for commercial application where highly non-normally distributed features like money spent are important may not be the best test. Nevertheless, we will use it for now and define a simple utility function before we get on to the tests:

## Utility function to make plots of Boruta test results
make.plots <- function(b, num,
                       true.var = NA,
                       main = paste("Boruta feature selection for test", num)) {
    write.text <- function(b, true.var) {
        if ( !is.na(true.var) ) {
            text(1, max(attStats(b)$meanZ), pos = 4,
                 labels = paste("True vars are V.1-V.",
                     true.var, sep = ""))        
        }
    }
    plot(b, main = main, las = 3, xlab = "")
    write.text(b, true.var)
    png(paste(run.name, num, "png", sep = "."), width = 8, height = 8,
        units = "cm", res = 300, pointsize = 4)
    plot(b, main = main, lwd = 0.5, las = 3, xlab = "")
    write.text(b, true.var)
    dev.off()
}

Test 1: Simple test of single significant variable

For a simple classification based on a single variable, Boruta performs well: while it identifies three variables as being potentially important, this does include the true variable (V.1) and the plot clearly shows it as being by far the most significant.

## 1. Simple test of single variable
y.1 <- factor( ifelse( x$V.1 >= 0, 'A', 'B' ) )

b.1 <- Boruta(x, y.1, doTrace = 2)
make.plots(b.1, 1)

Test 2: Simple test of linear combination of variables

With a test of a linear combination of the first four variables where the weights are decreasing from 4 to 1, we begin to get closer to the limitations of the approach.

## 2. Simple test of linear combination
n.dep <- floor(n.var/5)
print(n.dep)

m <- diag(n.dep:1)

y.2 <- ifelse( rowSums(as.matrix(x[, 1:n.dep]) %*% m) >= 0, "A", "B" )
y.2 <- factor(y.2)

b.2 <- Boruta(x, y.2, doTrace = 2)
make.plots(b.2, 2, n.dep)

The implementation correctly identified the first three variables (with weights 4, 3, and 2, respectively) as being important, but it had the fourth variable as possible along with the two random variables V.8 and V.9. Still, six variables are more approachable than twenty.

Test 3: Simple test of less-linear combination of four variables

For this text and the following we consider less obvious combinations of the first four variables. If we just count how many of them are positive, then we get to a situation where Boruta excels (because random forests excel at this type of problem).

## 3. Simple test of less-linear combination
y.3 <- factor(rowSums(x[, 1:n.dep] >= 0))
print(summary(y.3))
b.3 <- Boruta(x, y.3, doTrace = 2)
print(b.3)
make.plots(b.3, 3, n.dep)

Test 4: Simple test of non-linear combination

For a spectacular fail of the Boruta approach we will have to consider a classification in the hyperplane of the four variables. For this simple example, we simply count if there are an even or odd number of positive values among the first four variables:

## 4. Simple test of non-linear combination
y.4 <- factor(rowSums(x[, 1:n.dep] >= 0) %% 2)
b.4 <- Boruta(x, y.4, doTrace = 2)
print(b.4)
make.plots(b.4, 4, n.dep)

Ouch. The package rejects the four known significant variables. It is too hard for the random forest approach. Increasing the number of observations to 1,000 does not help though at 5,000 observations Boruta identifies the four variables right.

Limitations

Some limitations of the Boruta package are worth highlighting:

1. It only works with classification (factor) target variables. I am not sure why: as far as I remember, the random forest algorithm also provides a variable significance score when it is used as a predictor, not just when it is run as a classifier.

2. It does not handle missing (NA) values at all. This is quite a problem when working with real data sets, and a shame as random forests are in principle very good at handling missing values. A simple re-write of the package using the party package instead of randomForest should be able to fix this issue.

3. It does not seem to be completely stable. I have crashed it on several real-world data sets and am working on a minimal set to send to the authors.

But this is a really promising approach, if somewhat slow on large sets. I will have a look at some real-world data in a future post.

@online{2010,
  author = {},
  title = {Feature Selection: {All-relevant} Selection with the {Boruta}
    Package},
  date = {2010-11-15},
  url = {https://www.cybaea.net/Journal/2010/11/15/Feature-selection-All-relevant-selection-with-the-Boruta-package.html},
  langid = {en-GB-oxendict}
}

However, if you want to replicate their analysis in standard R, then you can absolutely do so, and we show you how.

Data preparation

First you need to prepare the rather large data set that they use in the Revolutions white paper. The preparation script shown below does two passes over all the files which is not needed: changing it to a single pass is left as an exercise for the reader…. Note that the following script will take a while to run and will need some 30-odd gig of free disk space (another exercise: get rid of the airlines.csv file), but once it is done the analysis is fast.

## big.R - Preprocess the airline data
## Copyright © 2010 Allan Engelhardt (http://www.cybaea.net/)

## Install the packages we will use
install.packages("bigmemory",
                 dependencies = c("Depends", "Suggests", "Enhances"))

## Data sets are downloaded from the Data Expo '09 web site at
## http://stat-computing.org/dataexpo/2009/the-data.html
for (year in 1987:2008) {
    file.name <- paste(year, "csv.bz2", sep = ".")
    if ( !file.exists(file.name) ) {
        url.text <- paste("http://stat-computing.org/dataexpo/2009/",
                          year, ".csv.bz2", sep = "")
        cat("Downloading missing data file ", file.name, "\n", sep = "")
        download.file(url.text, file.name)
    }
}

## Read sample file to get column names and types
d <- read.csv("2008.csv.bz2")
integer.columns <- sapply(d, is.integer)
factor.columns  <- sapply(d, is.factor)
factor.levels   <- lapply(d[, factor.columns], levels)
n.rows <- 0L

## Process each file determining the factor levels
## TODO: Combine with next loop
for (year in 1987:2008) {
    file.name <- paste(year, "csv.bz2", sep = ".")
    cat("Processing ", file.name, "\n", sep = "")
    d <- read.csv(file.name)
    n.rows <- n.rows + NROW(d)
    new.levels <- lapply(d[, factor.columns], levels)
    for ( i in seq(1, length(factor.levels)) ) {
        factor.levels[[i]] <- c(factor.levels[[i]], new.levels[[i]])
    }
    rm(d)
}
save(integer.columns, factor.columns, factor.levels, file = "factors.RData")

## Now convert all factors to integers so we can create a bigmatrix of the data
col.classes <- rep("integer", length(integer.columns))
col.classes[factor.columns] <- "character"
cols  <- which(factor.columns)
first <- TRUE
csv.file <- "airlines.csv"   # Write combined integer-only data to this file
csv.con  <- file(csv.file, open = "w")

for (year in 1987:2008) {
    file.name <- paste(year, "csv.bz2", sep = ".")
    cat("Processing ", file.name, "\n", sep = "")
    d <- read.csv(file.name, colClasses = col.classes)
    ## Convert the strings to integers
    for ( i in seq(1, length(factor.levels)) ) {
        col <- cols[i]
        d[, col] <- match(d[, col], factor.levels[[i]])
    }
    write.table(d, file = csv.con, sep = ",", 
                row.names = FALSE, col.names = first)
    first <- FALSE
}
close(csv.con)

## Now convert to a big.matrix
library("bigmemory")
backing.file    <- "airlines.bin"
descriptor.file <- "airlines.des"
data <- read.big.matrix(csv.file, header = TRUE,
                        type = "integer",
                        backingfile = backing.file,
                        descriptorfile = descriptor.file,
                        extraCols = c("age"))

Sample analysis

All done now. Sample analysis:

## bigScale.R - Replicate the analysis from http://bit.ly/aTFXeN with normal R
##   http://info.revolutionanalytics.com/bigdata.html
## See big.R for the preprocessing of the data

## Load required libraries
library("biglm")
library("bigmemory")
library("biganalytics")
library("bigtabulate")

## Use parallel processing if available
## (Multicore is for "anything-but-Windows" platforms)
if ( require("multicore") ) {
    library("doMC")
    registerDoMC()
} else {
    warning("Consider registering a multi-core 'foreach' processor.")
}

day.names <- c("Monday", "Tuesday", "Wednesday", "Thursday", "Friday",
               "Saturday", "Sunday")

## Attach to the data
descriptor.file <- "airlines.des"
data <- attach.big.matrix(dget(descriptor.file))

## Replicate Table 5 in the Revolutions document:
## Table 5
t.5 <- bigtabulate(data,
                   ccols = "DayOfWeek",
                   summary.cols = "ArrDelay", summary.na.rm = TRUE)
## Pretty-fy the outout
stat.names <- dimnames(t.5.2$summary[[1]])[2][[1]]
t.5.p <- cbind(matrix(unlist(t.5$summary), byrow = TRUE,
                      nrow = length(t.5$summary),
                      ncol = length(stat.names),
                      dimnames = list(day.names, stat.names)),
               ValidObs = t.5$table)
print(t.5.p)
#             min  max     mean       sd    NAs ValidObs
# Monday    -1410 1879 6.669515 30.17812 385262 18136111
# Tuesday   -1426 2137 5.960421 29.06076 417965 18061938
# Wednesday -1405 2598 7.091502 30.37856 405286 18103222
# Thursday  -1395 2453 8.945047 32.30101 400077 18083800
# Friday    -1437 1808 9.606953 33.07271 384009 18091338
# Saturday  -1280 1942 4.187419 28.29972 298328 15915382
# Sunday    -1295 2461 6.525040 31.11353 296602 17143178

## Figure 1
plot(t.5.p[, "mean"], type = "l", ylab="Average arrival delay")

Just like the Revolutions paper. You can now use biglm.big.matrix and bigglm.big.matrix for basic regression and there are also k-means clustering and other functions.

I must admit here that I do not understand the Revolutions regression example, so I have not attempted to replicate it here. It seems kind of sad if they change the syntax to be incompatible with standard R formulas, which is what appears to be happening.

Credit to Michael Kane and Jay Emerson of Yale who showed much of this in their poster The Airline Data Set… What’s the big deal?.

The key technique is to draw a gradient line which R does not support natively so we have to roll our own code for that. Unfortunately, lines(..., type="l") does not recycle the colour col= argument, so we end up with rather more loops than I thought would be necessary.

(The answer is not to use lines(..., type="h") which, confusingly, does recycle the colour col= argument. This one had me for a while, but the type=h lines always start from zero so you do not get the gradient feature.)

We also get a nice opportunity to use the under-appreciated read.fwf function.

## nino.R - another version of http://bit.ly/9P9Gh1
## Copyright © 2010 Allan Engelhardt (http://www.cybaea.net/)

## Get the data from the NOAA server
nino <- read.fwf(
  "https://www.cpc.ncep.noaa.gov/data/indices/wksst.for",
  widths = c(-1, 9, rep(c(-5, 4, 4), 4)),
  skip = 4,
  col.names = c(
    "Week",
    paste(
      rep(
        c("Nino12", "Nino3", "Nino34", "Nino4"), rep(2, 4)
      ),
      c("SST", "SSTA"),
      sep = "."
    )
  )
)

## Make the date column something useful
nino$Week <- as.Date(nino$Week, format = "%d%b%Y")

## Make colour gradients
ncol <- 50
grad.neg <- hsv(4 / 6, seq(0, 1, length.out = ncol), 1) # Blue gradient
grad.pos <- hsv(0, seq(0, 1, length.out = ncol), 1) # Red gradient

## Make plot
{ ## <- This block to work in R Markdown / Quarto
  plot(
    Nino34.SSTA ~ Week,
    data = nino,
    type = "n",
    main = "Nino34",
    xlab = "Date",
    ylab = "SSTA",
    axes = FALSE
  )
  do.call(
    function(...) {
      rect(..., col = "gray85", border = NA)
    },
    as.list(par("usr")[c(1, 3, 2, 4)])
  )

  y <- nino$Nino34.SSTA # The values we will plot
  x <- nino$Week

  axis.Date(1,
    x = x,
    tck = 1,
    col = "white"
  )
  axis(2, tck = 1, col = "white")
  box()

  idx <- integer(NROW(nino))
  idx[y >= 0] <-
    1 + round(y[y >= 0] * (ncol - 1) / max(y[y >= 0]), 0)
  idx[y < 0] <-
    1 + round(-y[y < 0] * (ncol - 1) / max(-y[y < 0]), 0)

  draw.gradient <- function(x, ys, cols) {
    xs <- rep(x, 2)
    for (i in seq(1, length(ys) - 1)) {
      plot.xy(list(x = xs, y = c(ys[i], ys[i + 1])),
        type = "l", col =
          cols[i]
      )
    }
  }

  for (i in 1:length(x)) {
    ys <- seq(0, y[i], length.out = idx[i] + 1)
    cols <- (if (y[i] >= 0) {
      grad.pos
    } else {
      grad.neg
    })
    draw.gradient(x[i], ys, cols)
  }
}

The result is a decent gradient:

I deliberately omitted the scale legend on the right hand side following Allan’s First Law of Happy Graphics: Thou shall not present the same information twice.

For less dense information, you should increase the line width. That is left to the reader. (Hint: it is hard to get just right in base graphics, but lwd <- ceiling(par("pin")[1] / dev.size("in")[1] * dev.size("px")[1] / length(x)) could be a starting point for an approximation. We really need gradient-filled polygons in base R.)

Let’s try the FTSE-100 index of leading UK companies to see if they are significantly different from the S&P 500 leading American companies that we analyzed four years ago.

We will of course use the R statistical computing and analysis platform for our analysis, and once again we are grateful to Yahoo Finance for providing the data.