A/B testing. Is Khan doing it wrong?

A/B testing is where you try an old way of doing things and a new way each with a sample of the users and see which one works better. It is frequently used by ads where you test two wordings "Buy new coke!" and "Buy improved coke!" to see which one gets more clicks.

If you ever write a book or need to name a shop you should spend a few quid buying google ads for the two names you are trying to pick "How to start a fight" and "How to win an argument" and see which one gets clicked on more. That one should be the name.

This is one of those embarrasing posts that is probably wrong. But I think the A/B testing used by the Khan academy makes a fairly fundamental mistake. The Khan acedemy has lessons in various subjects. So presumably they want to use A/B testing to see if kids taught "1+1=2" or "1 + 1 = 2" learn more quickly and such.

A/B testing is a useful way to see if little tweaks result in better user experience. In Khan's case learning. It does not substitute for good design vision but can help make some relatively small tweaks. Improving the Khan acedemy and kids education is really important so if there is a bug in their A/B testing they might be making the wrong choices about how to improve their teaching.

For this kind of testing you need to pick the number of test cases in advance. How not to run an A/B test explains why and the effects of looking before all the test is finished. This is an odd feature of frequentist statistics

'However, the significance calculation makes a critical assumption that you have probably violated without even realizing it: that the sample size was fixed in advance. If instead of deciding ahead of time, “this experiment will collect exactly 1,000 observations,” you say, “we’ll run it until we see a significant difference,” all the reported significance levels become meaningless. This result is completely counterintuitive and all the A/B testing packages out there ignore it'

The A/B testing used by Khan seems not to do this as the gae bingo system says

"Controlling and ending your experiments

Typically, ending an experiment will go something like this:

You'll notice a clear experiment winner and click "End experiment, picking this" on the dashboard. All users will now see your chosen alternative."



This seems to be saying that either you should notice what is statistically significant which you won't always or that something can be declared statistically significant before all the samples are tested. Think of it this way. If every test has a 5% chance of being wrong and you think of everytime you look at the A/B test as adding 5% to the chance of being wrong. It is not quite that bad but it gives you a feeling of the problem.

Now there are ways you can tell that something is statistically significant really early in a test. "Bayesian Statistics and the Efficiency and Ethics of Clinical Trials" deals with these. In medical trials you want to know as early as possible if a new treatement is better or worse than an old one. Giving someone the wrong ad wont kill anyone but the wrong cancer drug might. This paper goes through how you would figure this out using Bayesian methods. These methods are also described in chapter 37 of MacKay's 'Information Theory, Inference, and Learning Algorithms'

But looking at the code GAE bingo uses for A/B testing they do not seem to be using these methods. So it looks to me that they are making the mistake of letting you stop a test when you want to. Which in frequentist statistics can be an error.

Also I think Vanity another rails A/B testing framework makes the same assumption
"This experiment will conclude once it has 1000 participants for each alternative, or a leading alternative with probability of 95% or higher:"

The system used by the BBC is based on time and not numbers according to this article. "Example use
For 5 in 100 people to get a two-option test running for 24 hours the function is initialised like this:". Which is not nearly as bad. But it is assuming that at the end of the time period you have had enough users to make a good test.

There is a proper explanation as to what can happen if you stop a trial early in "How not to run A/B testing". But from my reading many of the A/B testing frameworks out there seem to be making this error. Please correct me in the comments.

Addition: Ben from gae bingo got back to me in a comment on their blog.
"You're right that this is an issue, and that's a great blog post. However, this is significantly mitigated by a) letting your experiments run long enough to get a large-ish sample size for your population and b) simply not checking your dashboard constantly and making snap decisions.

We could build stuff into the system to mandate that, but at the moment I believe we'll be able to get solid value out of the existing framework (just like most A/B systems)." This seems fair enough, Khan are going to have such a high volume of users that they will be able to get a large sample size quickly.

Allen Downey has a great review of the problem with simulations here

War is on its last legs

Famine is not looking too great either. Pestilence and death are down but not nearly out.

The recent riots in London had all sorts of people claiming the world had gone Mad Max and it was time to shut down facebook and make all teenagers join the army. There is a great summary of the moral panic in this article "Civil disorder and looting hits Britain, We have been here before". I think its interesting how the rioting that probably wont even be visible in the end of the year crime statistics made everyone freak out so much.

For all the talk of moral decay is the world in general a worse place than it used to be? War along with the other horsemen is a pretty good definition of how bad things are. Unless Bob Geldof finally gets that concert to end death set up. Even pestilence would seem to be a big ask to fix. What is the trend for war deaths recently?

Many people have predicted a century of appalling death tolls from war. Robert McNamara the ex US defense secretary and self confessed war criminal (at 3:10 ) for example.

Blight and McNamara project the level of warfare forward into the twenty- first century based on population growth, and suggest a “speculative” but “conservative” estimate of “at least 300 million” fatalities from war in the twenty-first century, of which perhaps 75 million would be military

So how have we done in the last decade on the predicted 3 million deaths a year?

So far they haven't even been close. In fact, the last decade has seen fewer war deaths than any decade in the past 100 years, based on data compiled by researchers Bethany Lacina and Nils Petter Gleditsch of the Peace Research Institute Oslo. Worldwide, deaths caused directly by war-related violence in the new century have averaged about 55,000 per year, just over half of what they were in the 1990s (100,000 a year), a third of what they were during the Cold War (180,000 a year from 1950 to 1989), and a hundredth of what they were in World War II.

So there has been under 2% of the expected war deaths this decade.

The Quaker economist describes the decline with "Just in the last fifty years, the progress towards peace has been startling. If this trend continues — and I am cautiously optimistic that it will — then as pressure from civil society increases and international institutions of conflict resolution mature, warfare as a regular practice may cease sometime late this century."

While talking about sending the four apocalyptic horsemen to their dotage the decline of Famine is also impressive

On the day that a famine is declared in Africa -- thanks as much to Somalia's political dysfunctionality as to a severe drought -- comes news that the world drought severity index has been falling for three decades. You can check for yourself by going here and clicking on "make time series":

There are some fairly serious disclaimers on this famine decline. The first is that agricultural productivity growth is declining at the moment
combine this with a possible loss of farm productivity when global warming kicks in and there could be problems.

While on the subject of how things are not good but significantly better than they used to be the last decade was the best one in human history.

We estimate that between 2005 and 2010, nearly half a billion people escaped extreme hardship, as the total number of the world's poor fell to 878 million people.

Nearly everything is getting better. Including life expectancy which you could take as evidence that even death is not what he was. Oddly all three of these sources give different numbers on 'major armed conflicts' but all agree the number is declining.

If two of the biggest problems mankind has faced for all of history are on the decline so rapidly they could be almost unknown within our life times I would say that is good news. And probably more important evidence of how we are progressing than a few rioters stealing runners.

We are getting better at nearly everything

How quickly are humans getting better? We tend to think technology is getting better or that humans augmented by technology are improving. New swimming records happen regularly as swim suit technology improves. This post just throws up some evidence about human progress.

The Effect of Testing for Performance Enhancing Drugs on the Progress of World Records in Weightlifting “From 1964 to 1988 the relative strength of the world record holders in those weight classes increased by 21% …The same analysis in other types of sports, where there had been some changes in training methods over the same period of time, revealed that the maximum improvement was only 9%“ So most improvement in weightlifting seems to have been from pharmacological rather than having a wider range of people to select from or improved training mechanism reasons. However what about areas of human endeavour that drug taking seems unlikely to help?

In chess taking steroids seems unlikely to help. Though future nootropics might. "We conclude that there has been little or no ‘inflation’ in ratings over time—if anything there has been deflation. This runs counter to conventional wisdom, but is predicted by population models on which rating systems have been based…The results also support a no answer to question 2. In the 1970’s there were only two players with ratings over 2700, namely Bobby Fischer and Anatoly Karpov, and there were years as late as 1981 when no one had a rating over 2700 (see [Wee00]). In the past decade there have usually been thirty or more players with such ratings."

Even musicians are getting better Virtuosos Becoming a Dime a Dozen "The overall level of technical proficiency in instrumental playing, especially on the piano, has increased steadily over time." One good explanation for this and the chess improvement is just that more people are getting to try these, people who are not as limited by nutrition and disease as they would have been int he past.


This explanation is expanded in this rather good post "Two Hour Marathon in 2045"

"But the pipeline that selects and trains runners behaves, in some ways, like the model. If a person with record-breaking potential is born in Kenya, where running is the national sport, the chances are good that he will be found, he will have opportunities to train, and he will become a world-class runner. It is not a certainty, but the chances are good.

If the same person is born in rural India, he may not have the opportunity to train; if he is in the United States, he might have options that are more appealing.

So in some sense the relevant population is not the world, but the people who are likely to become professional runners, given the talent. As long as this population is growing exponentially, world records will increase linearly.

That said, the slope of the line depends on the parameter of exponential growth. If economic development increases the fraction of people in the world who have the opportunity to become professional runners, these curves could accelerate."

Progress of things like number of children suffering malnutrition and having clean water can really result in increasing the number of great chess or piano players as well as the world running record. We are getting better at loads of things because more people are getting to try them without the poverty induced hinderances they used to have. According to this model it is not that population is increasing exponentially but at the moment population of people who have a chance at being great at something is.

If you have other explanations for why and in what human achievment is progressing I would love to hear them.

Do Budget Cuts and Riots Go Together?

I saw this paper 'Austerity and Anarchy: Budget Cuts and Social Unrest in Europe, 1919-2009'
with the intro
'In the wake of this week's London riots, some commentators have linked the youth unrest to budget cuts. The authors of CEPR DP8513 explore the historical basis for this view and finds that austerity and violence have tended to go hand in hand.'

I am reading through the paper now. But I wonder if there was an image that could quickly show a connection between the two.

Wikipedia lists the riots since 1950 as

1958 Notting Hill race riots
1970 Garden House riot
1971 Priestlley riots
1975 Chapeltown race riot
1977 Battle of Lewisham
1980 St. Pauls riot
1981 England riots · Brixton riot · Chapeltown race riot · Toxteth riots · Moss Side riot · Handsworth race riots
1985 Brixton riot · Broadwater Farm riot
1987 Chapeltown race riot
1989 Dewsbury race riot
1990 Strangeways Prison riot · Poll Tax riots
1991 Meadow Well riots
1995 Manningham riot · Brixton riot
2001 Bradford riots · England riots · Oldham race riots · Harehills riot
2005 Birmingham race riots
2010 UK student protests
2011 London riots

Now if you look at the government spending as a percentage of GDP here. On top of this graph I put a bar for every riot each year one occurred.

Counting all riots as the same is not fair. Their graph goes from 34->48 whereas the riots go from 0->6. Laying the first on the second is not considered good practice in data visualisation. Also and this is a big one. If GDP drops as in a recession and the percentage of government spending to GDP stays the same total government spending will drop. a fairer graph would look at gdp or government spending adjusted compared to riots not the two combined.

The proper paper says rioting and austerity go hand in hand. I will read it carefully to see how close the link is. But a quick look at the data and no obvious major link jumps out at me.

We have reached Peak Baby

When will we reach peak babies? In what year will the most children be born? I bet last night a shiny pint that we will reach peak babies in the next three years. That the most babies ever born will be in a year before 2015.

Now I accept that we could never actually know how many children will be born in the future. The bet will end when I present enough evidence to convince those I am betting with rather than with a proof. Demographics is regarded as one of the most predictable of social sciences but some possible future invention could drastically increase the number of babies. We could have brave new world style artificial wombs of some such that vastly increases the birth rate for example.

Hans Rosling the statitician tweeted recently.

I looked up the UN data on this here. In detailed indicators look in births and in select country look in world. The highest birth number in the world was 1985-1990
Period Births per year
1950-1955 97 769
1955-1960 102 894
1960-1965 110 280
1965-1970 118 200
1970-1975 121 715
1975-1980 120 676
1980-1985 129 088
1985-1990 137 207
1990-1995 134 960
1995-2000 132 473
2000-2005 131 644
2005-2010 134 072
2010-2015 135 775
2015-2020 135 396
2020-2025 133 800
2025-2030 132 452
2030-2035 131 991
2035-2040 132 099
2040-2045 131 926
2045-2050 131 127
2050-2055 129 904
2055-2060 128 785
2060-2065 127 998
2065-2070 127 402
2070-2075 126 725
2075-2080 125 823
2080-2085 124 775
2085-2090 123 753
2090-2095 122 837
2095-2100 121 992
Now it could all of a sudden rise up and any prediction for the future is unlikely to be as accurate as historical estimates. Still I think I win the bet.

This does not mean that population will drop. As we are living longer world population is still expected to grow. But it does mean we can say that Malthus was wrong. Human numbers will not grow exponentially barring some disaster.

Crowd Sourced Optimal Fantasy Football Team

All my fantasy football attempts have the problem that I know nothing about football. So for example Berbatov is unlikely to do as well this season as last season so picking him would seem unwise. But how does someone who knows nothing about football find out who will play well next season?

The wisdom of the crowds is the James Surowiecki about "the aggregation of information in groups, resulting in decisions that, he argues, are often better than could have been made by any single member of the group." This sort of thing does not work if the crowd has a bias in a particular direction.

I decided to look at the people is the "team selected by %" that fantasy premier league shows you. If I rerun the optimisation described in this post but instead of trying to great a team that has the maximum number of points last season I try and get the team whose players have been selected by the most other fantasy football managers.

The idea is that a team with the right number of defenders, goalkeepers, midfielders and forwards, that has at most three players from one team, that costs less than 100 and whose members have been picked most often should be really good.

The most popular team is

      Player Club Pos Price Pts people
7     Al-Habsi  WIG  GK    45 125  199410
30        Bale  TOT MID    80 118   189087
37      Barton  NEW MID    60 131   123865
69      Cahill  BOL DEF    55 105   180751
184      Given  AVL  GK    50   0   215091
205  Hangeland  FUL DEF    65 154   192333
231       Huth  STO DEF    60 138   165684
266    Kompany  MCI DEF    60  95  197352
323      N'Gog  LIV STR    55  48   279932
334 Odemwingie  WBA STR    75 171   218485
421     Suarez  LIV STR    95  68   343361
424    Taarabt  QPR MID    65   0   141086
452      Vidic  MUN DEF    80 148   172773
472   Wilshere  ARS MID    65  93   220381
477 Yaya Toure  MCI MID    80 146   167706

Some of these players were probably picked for the first few games and will be transfered out when they are about to play tougher games.

The average player is picked 26794 times. These players have been picked 200486 people on average.
This team would have scored 1415 points last season. The best team I could have picked for last season would have scored 2332 points. I have updated the dataset to include this people picked statistic.

Fantasy Football Optimization 2

So it turns out I was thick last post about picking the fantasy football team. As my friend Bren explained to me "you choose 11 players to be "on the pitch" each week, plus a captain who scores double (and a vice captain in case the captain doesn't play). can only have 1 keeper in the first 11, can choose any formation with a minimum of 3 defenders, 3 midfielders and 1 forward."

So the problem is now pick 11 players that will actually play and some really cheap players that won't

So whats the cheapest of each player you could get?
The cheaperst players in each position are about 4 or 4.5 each. For example
Goalkeeper
Moreira SWA 4.0 0
Defenders
Whitbread NOR 4.0 0
Tate SWA 4.0 0
Midfield
Lappin NOR 4.5 0
Gecov FUL 4.5 0
Allen SWA 4.5 0
Strikers
Miller WBA 4.5 6
Hulse QPR 4.5 0
Agyemang QPR 4.5 0

So if you picked from this price of player knowing they would not play that would allow you to spend more money on those that were on the pitch. Now defenders score less points than midfielders and strikers so lets say we only want three of them. And we then buy two cheap ones that we dont play. 2 cheap defenders costs 8 and one goalie for 4 means we now have 88 to pick the remaining 12 players. The case whether you should have 5 midfielders or 3 strikers is less clear. I will try both and see which has a higher score.

Attempt 1. Cheap goalie and 2 defenders and 1 midfielder. Cost 16.5 on 4 non playing players. Trying to have 3 strikers. The constraints would now be

num_goalkeepers <- 1
num_defenders <- 3
num_midfielders <- 4
num_strikers <- 3
max_team_cost <- 835
max_player_from_a_team <- 3 

Gives a team of

1         Hart  MCI  GK    70 175
48    Ivanovic  CHE DEF    70 144
49        Huth  STO DEF    60 138
51      Hughes  FUL DEF    50 129
197       Adam  LIV MID    90 192
198    Malouda  CHE MID   105 186
200    Dempsey  FUL MID    85 168
201   N'Zogbia  WIG MID    75 167
210     Jarvis  WOL MID    60 133
386   Berbatov  MUN STR    95 176
388 Odemwingie  WBA STR    75 171

scoring 1779. You would make Adam your captain and he would have double points.

Attempt 2. Cheap non playing goalie and 2 defenders and 1 striker. Trying to have two strikers. Cost 16.5 on 4 non playing players.

num_goalkeepers <- 1
num_defenders <- 3
num_midfielders <- 5
num_strikers <- 2
max_team_cost <- 835
max_player_from_a_team <- 3 

1         Hart  MCI  GK    70 175
48    Ivanovic  CHE DEF    70 144
49        Huth  STO DEF    60 138
51      Hughes  FUL DEF    50 129
197       Adam  LIV MID    90 192
198    Malouda  CHE MID   105 186
200    Dempsey  FUL MID    85 168
201   N'Zogbia  WIG MID    75 167
210     Jarvis  WOL MID    60 133
386   Berbatov  MUN STR    95 176
388 Odemwingie  WBA STR    75 171

with a score of 1769. Again Adam would be your captain. It looks like three strikers is a better plan than 5 midfielders. But it is a close run thing.

Just to make things really complicated Bren explained that you should generally have one good player on the subs bench in case one of the rest of the team is injured. "it's quite common for one of the main team to miss a week so you may need to make 1 sub an excellent player (probably defender since they're cheaper)". Having a defender as your good sub has the advantage that you are allowed to play with one forward and three midfielders so if one of your three forwards, four midfielders or three playing defenders gets injured a defender can sub in for any of those.

So assuming we actually want four defenders. We would have one cheap goalie and one cheap defender and one cheap midfielder. At a cost of 12.5 for non playing players.

num_goalkeepers <- 1
num_defenders <- 4
num_midfielders <- 4
num_strikers <- 3
max_team_cost <- 875
max_player_from_a_team <- 3 

giving a team of

         Player Club Pos Price Pts
1         Hart  MCI  GK    70 175
46      Cole A  CHE DEF    75 150
49        Huth  STO DEF    60 138
51      Hughes  FUL DEF    50 129
54    Bardsley  SUN DEF    50 123
197       Adam  LIV MID    90 192
200    Dempsey  FUL MID    85 168
201   N'Zogbia  WIG MID    75 167
203    Downing  LIV MID    85 163
386   Berbatov  MUN STR    95 176
388 Odemwingie  WBA STR    75 171
393   Davies K  BOL STR    65 132

1884 points -123 as Bardley wont play = 1761. Basically Bardsley a Sunderland defender is really cheap at 5. This is one more than you pay for a player you don't want to play but if you do need to lay him he gets 123 points a season.
Davies looks a little low there with 132 points. The worst midfielder has 163 points. So what if we get rid of the third striker and try 5 midfielders?

num_goalkeepers <- 1
num_defenders <- 4
num_midfielders <- 5
num_strikers <- 2
max_team_cost <- 875
max_player_from_a_team <- 3 

   Player Club Pos Price Pts
1         Hart  MCI  GK    70 175
46      Cole A  CHE DEF    75 150
49        Huth  STO DEF    60 138
51      Hughes  FUL DEF    50 129
53      Distin  EVE DEF    55 124
197       Adam  LIV MID    90 192
200    Dempsey  FUL MID    85 168
201   N'Zogbia  WIG MID    75 167
203    Downing  LIV MID    85 163
210     Jarvis  WOL MID    60 133
386   Berbatov  MUN STR    95 176
388 Odemwingie  WBA STR    75 171

1886-124 for Distin who usually wont play = 1762. So this is my best single fantasy football team of last season.

The more I learn about this game the more nuances it has. Off the top of my head
1. Model that the captain gets double points. In this case Adam the highest scoring player is actually quite cheap. But in the case where he was really expensive you would want to take into account that he can earn double points.
2. Take into account who teams are playing. With a full game by game scoring dataset you can investigate really interesting patterns like if playing top five club means less points and bottom five more. And if so make transfers based on upcoming games. This seems to be where a huge amount of the skill in fantasy football is.
3. Change the code so you can have 3->5 defenders, 3-5 midfielders and 1-3 strikers but only 11 total players.
4. Just picking cheap non playing players is probably wrong. You at least want to pick the best cheap players you can. Which I have not. I am told Shane Ferguson is a good buy so I will probably make him one of the cheap players.

If any of the intuitions about having a good substitute or any of my other assumptions are wrong please correct me.
If you predict different points for players this season. If you want to try this method for next season but dont want to run the program linked to in part one


1. Copy the dataset I have here
2. Put this data into a new google docs spreadsheet.
3. Make your predictions on the number of points they will score. So if you think Berbatov won't score 176 points this season but only 160 change that points value. You can delete players you are not interested in as well.
4. Put your new spreadsheet URL in the comments
and I will run a optimization over your predictions for you.

Fantasy Football Optimisation

Some friends challenged me to a game called fantasy "football" where you pretend to be a Russian billionaire. Not the bit where you steal natural resources off the population but where you buy a bunch of poncy overpaid foreigners who flounce around and earn insane amounts of money.

While I'm ranting about "football". Why do football ads always say it has "the beauty of a dance". If you like dancing that much go to the ballet



Anyway So I have to pretend to know something about this "football" which I dont but I do know a bit about optimization and more about copying stuff. I remembered this old R package article about optimising for fantasy football. This was written by prasoonsharma and I have just reused his code.

I scraped last seasons scores off the fantasy premierleague website. Some parsing turned this into the correct format. You can download the cleaned up data here

The constraints for this version of the game are slightly different than the one prasoonsharma was playing. This means you need more goalies, have more money to spend on players and other such changes.

The best single team for last season would have been

        Player Club Pos Price Pts
1         Hart  MCI  GK    70 175
5     Al-Habsi  WIG  GK    45 125
48    Ivanovic  CHE DEF    70 144
49        Huth  STO DEF    60 138
51      Hughes  FUL DEF    50 129
54    Bardsley  SUN DEF    50 123
55     Johnson  WOL DEF    50 120
197       Adam  LIV MID    90 192
200    Dempsey  FUL MID    85 168
201   N'Zogbia  WIG MID    75 167
210     Jarvis  WOL MID    60 133
212     Barton  NEW MID    60 131
386   Berbatov  MUN STR    95 176
388 Odemwingie  WBA STR    75 171
393   Davies K  BOL STR    65 132

with a total score of 2224 points. You get to change the team every week in the real game. This optimisation is just if you got to pick one team and leave it.

Picking a fantasy football team involves
1. Predicting how many points each player will score that week
2. Optimising based on this prediction so you pick the team that covers all the rule constraints that scores the most points.

This code carries out the second part. If you have a prediction for how many points players will score that is better than
"exactly the same amount they scored last season" you can change the data with your new prediction and run it (or I can run it for you if you want). A web app that allowed you to enter your player score predictions and ran an optimisation so that the best team that could be picked based on your predictions was then created might be useful. If you save the data in a new google doc and change the Pts values to your player predictions. Then post a link to that google doc in the comments I will run the optimisation for you.

The Kilbrittain Whale

I love weird crap. I have loads of holiday photos of my wife looking bemused outside some odd tourist destination. Here is one of my wife outside the udderly brilliant Cork Butter Museum last Sunday.

But last week I think I may have reached the Zenith of weird tourist destinations. This acme of travel strangeness is the Kilbrittain Finn whale. This specimen of the second largest species of whale is mounted in the middle of a town several kilometers inland.

How did it get here? The Channel 4 whale autopsy provides some of the answers about this poor animals beaching in 2009. Three minutes in (or 20 seconds in the video below) you see the frankly unbelievable result of a bad autopsy of a whale. Exploding entrails all over the Denmark beach is the sort of image that stays with you.

 


What this pathological investigation misses though is the Machiavellian intrigue that lead to this particular resting place for the whale. When it landed in 2009 various locals tried to make off with the 50 ton pile of putrifying Balaenoptera physalus. "Apparently, an attempt was made on Saturday night under cover of darkness to steal the jaws but their bulk proved too much for the perpetrators. "

There are stories of a beach based stand off between the communities of Kilbrittain and Courtmacsherry. "Shortly after it died two men from Courtmacsherry, also known as a drinking village with a fishing problem, approached with a chainsaw. Their plan was to saw off its head and mount the jaw bones in the village. However, they were confronted by Kilbrittain residents who told them they were claiming the carcass." Presumably at some point in these tense negotiations two large groups of adults both armed with heavy machinery both realised that they were fighting for control of several dozen tons of rotting whale carcass.

There is no simile that really can convey the logic here. No metaphor can quite sum up the Irish ability to fight over possession of something that no one would want. But if I had go for one description it would be "like a stand off over 50 tons of festering cetacean".

Now having 'won' the whale, some lads from the local abattoir spent 18 months rendering the stinking baleen bag. This year and a half long job is shown in a frankly horrifying pictographic montage beside the the spoiled beluga that now rests in the middle of Kilbrittain park.

There was a great moment on redneck roundup (as nationwide is known inside RTE) about the whale where one of the guys who spent the lifespan of a hamster hacking at the pile of rancid blubber admits they may have gotten the order of some of the bones wrong.

One thing words cannot convey is the stench coming off this attraction. Some flesh still seems to cling to the head parts resulting in an overpowering odour of death.

It was so bad we could not eat the ice-creams we brought to the park. I advise you to go see the Kilbrittain whale while it still has this stench. There is nothing like the smell of dead whale to make you wonder about what some people will go see/fight over.

Really Old Idea

Sports really annoys me. Particularly at the end when they get boring. When they know they have a minute to go in football they start kicking the ball between them and generally acting the micky. I thought I was dead clever when I thought that they should have random end times of sports. If you do not know when the game will end it is harder to try run down the clock.

Axelrod had the idea of random endings years ago. Many other game theorists have had similar ideas at various times.

Then I heard that random endings have a much longer history. Adapt the new book from Tim Harford explains one really old example. This century IBM got a patent for "a “smooth-finish” auction, an auction that is programmed to end at a random time". This gets over the problem on ebay where at the end of an auction people try not to bid what they value the item at but at just enough to win the auction just before it closes. So instead of people bidding one cent over the current bid with one second left on the auction.

Paul Klemperer pointed out that this idea is quite old. Even though the is a US patent for it.

In a candle auction, the end of the auction is signaled by the expiration of a candle flame, which was intended to ensure that no one could know exactly when the auction would end and make a last-second bid. Sometimes, other unpredictable processes, such as a footrace, were used in place of the expiration of a candle.

Pepys wrote about such an auction in the 1660's
Auction by candle was known in England by 1641, when it is mentioned in the records of the House of Lords.[6] The practice rapidly became popular, and in 1652, John Milton wrote, "The Council thinks it meet to propose the way of selling by inch of candle, as being the most probable means to procure the true value of the goods."

This 350 years old idea is my current record on simple ideas that someone has beat me to. Hartford's book Adapt is full of these sorts of interesting stories. This extract about the Spitfire gives you a good feel for the book.

Malaria Modeling

There are two very interesting passages in Shah's 'The Fever' that suggest modeling of malaria can have an important role

HIV-positive people are most infective to others when the levels of virus in their bodies are high...Malaria triggers such spikes. Malaria infection, by inducing HIV to replicate, increases the viral load in HIV infected people. According to mathematical models, HIV may be responsible for 980,000 episodes of malaria and malaria responsible for mare than 8,000 Hiv infections in a single district of Kenya. The global effect of the malevolent partnership has yet to be mapped

This paper is the source of this information.

Another interesting point seems to suggest that a scapegoat that is more attractive to mosquitoes than people are.

the greater availability of cow and hog flesh attracted the interest of malaria mosquitoes.Hovering between home and stable ninety-nine out of one hundred times. That bit of anopheline caprice cost Plasmodeum dearly. As cows and sheep sprouted across the English countryside, malaria transmissionground to a halt

Increasing livestock numbers has serious effects on any ecosystem. the pig under the bad scheme should not be promoted without serious analysis. Which leads to a more general problem with complex problems like malaria. I want to talk about these issues in another post. There is an interesting Ted talk on malaria here. A lot of the ideas are impractical which is another reason to talk about practicality later.

Jeffrey Sacks also talks about the important role of livestock that distract mosquitoes in the end of poverty

Another important point is that some types of mosquitoes prefer to bite people, whereas others feed off cattle. Transmitting malaria requires two consecutive human bites: the first for the mosquito to ingest the parasite and the second for the mosquito to infect another person, roughly two weeks later. If the mosquito feeds frequently on cattle rather than on people, the odds are that at least one of the bites, if not both, will be taken from cattle. In India, for example, the predominant type of anopheles tends to bite humans about one third of the time, and cattle the rest. Africa, sadly has another predominating mosquito type which prefers human biting nearly 100 per cent of the time. Mathematically, the chance that an Indian mosquito will feed off two humans in a row is about one in nine, whereas in Africa it's about one in one

While on the subject of development I thought this quote was fascinating Justin Kilcullen head of Troicaire 48:18 in the Frontline 30/05/11 here
"Development aid is what we give to the developing world because we do not give them a decent deal in trade, because we have screwed them on their debts, because the flow of resources...lets change the economic structures that keeps these people poor thats the long term answer". If the head of an aid agency does not think they are the long term answer for development then they are probably not.

For the moment I am going to examine OpenMalaria to see how it can model these issues.

Coding Day Dublin

Can you do anything useful in a day? This is the premise of code day where computer programmers are invited to work with scientists for a day to try solve a real problem the scientists are trying to deal with. "For those unfamiliar with a code retreat, it’s a hack day with a focus, the angle placed solidly on solving interesting technical problems." Hacking here means writing short programs that do something cool.

On the 28th of May I spent a day in the Science gallery trying to create fixes for various small problems. There were about two dozen people there with about a third of the scientists. Declan, Andrea and Qamir who organise loads of these programmers meet ups around Dublin ran the event and helped make it a really friendly atmosphere.

The small projects that seemed to work best involved visualisation. The problem of taking data and presenting it in an interesting way seems just right for a few hours timeframe. Possibly this is because rather than a complex scientific issue all that has to be understood is what sort of information needs to be conveyed.

It is quite difficult to get up to speed on a problem in such a short period of time. I had a really interesting talk with Scrazzl but even understanding their system took so long I felt like I was wasting their time. This meant we spent a lot of time just understanding the basic background of someones research. Treemetrics were also doing some really interesting Operations research work on forest management. It was really interesting to hear what people were working on but perhaps a day that was focused on one theme might allow the coders to read up and get some understanding in advance. Not that I think all such coder days should have a single theme just that it might be interesting to try one that did.

One of the most interesting projects at the day was fight malaria at home. This is a distributed computer project like SETI at home or folding at home. The aim is to test drug molecules against proteins present in malaria to check for potential binds. This could than guide researchers as to what drugs are worth testing in real life. The guys from the project were really friendly and interesting. If you do have knowledge of BOINC or even just want to learn it it would be a great project to help out on.

Instead of just criticising are very fun and worthwhile day I will give an example of the sort of focused task I am talking about.

Something that is big, widespread and multidisciplinary. I am reading The Fever by Shah at the moment which made me think Malaria was one such problem. Off the top of my head diarrhea diseases, political expenses, maths education and are all areas that people from many areas could come together and do some interesting projects/demonstrations over a day on. But malaria is one of those big thorny problems that a hackday about could be really cool. I have a Google Document of possible ideas here

Bad Taxidermy

I have had a frankly morbid interest in taxidermy since I was a child. I particularly like bad taxidermy.

My absolute favorite form of taxidermy is when naturalists in the past attempted to reconstruct animals having never seen them. The weird mistakes they made should make raise serious epistemological questions. If animals we know that are quite similar to ones we see are gotten wrong how wrong are our reconstructions of Dinosaurs? More generally miscontructed taxidermy shows how our our ignorance and presumptions can distort our world.

My competition for the best misconstructed animal is between the Horniman walrus and King Fredericks lion.

With the Horniman walrus naturalist were sent the skin of the animal. Not realising it had wrinkles just kept stuffing the skin until it bloated up like some sort of zeppelin.


And the lion of King Frederick I of Sweden in 1731 where they had no idea what a lion looked like. The reconstruction has these weird old persons false teeth and lolcat eyes.


Paris for some unknown reason has particularly excellent taxidermy using shops

1. Deyrolle on rue du Bac.
2. Aurouze which they used in Ratatouille
3. The catacombs these freaked the bejasus out of me.

While I am on the subject I also recommend 'Stuff the World' a documentary on the world taxidermy championships.

The best crap taxidermy animatronics I have seen. And it is not a big category was in Strontium in Scotland. The wildlife museum there has a tunnel you crawl through where various crap animals on rails roll toward you. Strontium was on our travel list of places elements were discovered but that is a whole other blogpost.


If you also like appalling stuffed dead animals "crappy taxidermy" blog is a goldmine. If you know of any animals stuffed by people who have never seen the animal please post it in the comments

Spongebob New Wave Pants

I was watching Spongebob as usual last Saturday morning and I noticed how most of the characters were modeled on 70's rock musicians. Elvis Costello is a dead ringer for the eponymous hero and Brian Eno could take the wrap of a Squidwort crime if he ever took part in a line up.




Do you know any other 70's rock star look a likes?

How much is technology worth?

How much would you have to be paid to give up your mobile and your computer? How much is reading these pearls of wisdom, viewing pictures of amusing cats and forum fighting with idiots worth to you?

The brilliant Barking up the Wrong Tree blog it looks like mobiles are worth about 10% of income.

Keeping life satisfaction constant, we subsequently derive substantial GDP per capita estimates equivalent to a 10 percentage point increase in broadband and mobile phone penetration.

Median wage wage in US is about $32,000. So mobiles probably worth about 3 thousand dollars a year to the average American.

The value of the internet seems to vary from $1700 to $3800 according to this slate article

a model based on the value of people's time, ultimately estimating that Internet access is worth 2 percent of full income. Thus, they estimated the value of the welfare gain provided by the Internet ranged as high as $3,800 per person.

This means that in the 15 years that the internet and mobiles have become ubiquitous we are something like $5000 richer per year because of them.

Palaces of memory

Remembering is a creative process. We tend to think rote memorisation as a dull boring task. However instead we should think of it as "Moonwalking with Einstein". A big challenge to think of the oddest 'most memorable' images you can associate with what you are trying to remember.

The book 'Moonwalking with Einstein' deals with one journalists travels around the world of memory competitions. These competitions have all sorts of weird and wonderful rounds. Remembering binary number, packs of cards, poetry, random word list, matching faces to names and historic dates. It is a very entertaining book, dealing with the history of knowledge, savants, the weird world of self help marketing and the importance of living adventurously.


Many of these competition events seem pretty abstract. Some memory based skill that had some sort of humane self improvement aspect and maybe even some vague practical use would be good to learn though. But if there was one of these skills that would be cool to have I would like to practice the methods described in the book. I read a book, Inside the black room by Jack Allen Vernon, about sensory deprivation experiments in a Californian university. The author describes an Arab immigrant who unusually asked to be put back into sensory deprivation room as he wanted to train himself incase he was ever forced into a similar situation as torture.

Brian Keenan's 'an evil cradling' has vivid, gripping and terrifying descriptions of the madness that stalked him as he was held hostage in isolation in the Lebanon. For example he writes "that the human mind can travel into those dark regions and return exhausted but intact is more a miracle than that word can ever convey". One of the boys own adventure stories I read as a child had the description of an RAF POWs time in the cooler and the physical and mental regime he went through to ease the hardship. One of the parts I remember was that he remembered all the poetry he had learned at school. These tales of how to deal with sensory deprivation have really stayed with me. Maybe if the memory skills in this book could help in such a situation that would seem useful.

One of the expert mnemonists in the book Ed suggests memorising poetry and prose is useful for such a situation of sensory deprivation. "My philosophy of life is that a heroic person should be able to withstand about ten years in solitary confinement without getting terribly annoyed' he said 'an hour of memorization yields ten solid minutes of spoken poetry, and those ten minutes have enough content to keep you busy for a full day.'

I will give myself a challenge to get good at poetry memorisation in a month. The competition seems to be to '15 minute memorization time, 20 minute written recall' for poetry. I will try memorise a poem tomorrow. And again in a month and see if memory training improves poetry memorisation.

As an aside the author mentions the interesting possibility of using "neutropic 'cognitive steroids,'" but never goes into details about how these might aid a memory sports competitor. This is slightly odd given he wrote an article in 2005 (viewable here) about his experience taking one of these drugs. He said 'gym rats have steroids, and overachievers have Adderall. ' it would have been interesting to hear what effect these drugs had on his memory abilities but this book never brooches the subject in a personal way.

Search Engine Deoptimization

Take a company you have a problem with

1. Imagine you hate Ryanair. I do not btw they are just an example of a company this would work on.

2. You decide you want to harm them commercially

3. You go to their terms and conditions here and see they do not allow linking to their website

"Links to this website. You may not establish and/or operate links to this website without the prior written consent of Ryanair. Such consent may be withdrawn at any time at Ryanair’s own discretion."

There is a list of other companies that do not allow linking listed here. That where I heard of this T&C clause.

4. Now spider all the websites that do link to them. This is easy to do and you now should have a list of many websites. Either from a whois or just from scraping the website get an email address for each site.

5. Send all these linking websites official looking notices that remind them that "You may not establish and/or operate links to this website without the prior written consent of Ryanair" including a link to the official Ryanair terms and conditions. You would have to be careful here. You would not claim to be Ryanair for legal reasons. You do not want to break any laws or threaten anything. Just say you are from some official sounding company "John Smith and associates business compliance coordinators" and remind them of what the companies own policies say. It is not like you are lying.

6. Watch Ryanair drop down search rankings as people delete links to them. Search rankings are mainly based on how many (and what sort of) sites link to you. As they drop down the search engine rankings they will suffer commercially. With my google of "cheap flight dublin london" Ryanair comes up as the third site. Which implies they must get a fair amount of traffic from search engines.

I think this attack should be called Search Engine Deoptimization.

Analog Computer Museum

This week I was lucky enough to visit the Analog Computer Museum near Frankfurt. Bernd who owns and protects the collection I have known for years via email so I was happy out to finally get to meet him and his wife. These Analog computers carry out calculations using electronic components.


Bernd works as a software archeologist. That is he updates the systems of banks and such that are written in OS' and languages that almost no one knows anymore. If the software running your bank was written int he 60's and you want to get it running on Linux Bernd is the man to call. These languages are so obscure they make the COBOL look like the Beatles. The job seems to pay well enough as he can afford warehouses to keep his computer collection in and a collection of vintage sports cars he drives at terrifying speeds on the local autobahns.



Yes I take my wife to Computer Museums but in fairness shes a physicist and I suspect she thought this rocked as much as I did



These machines are ideal for real time simulation. This Telefunken currently simulating a car was used to train nuclear powerstation operators with simulations of various 'problems'


Next time you call rand() think of this board that was used to generate random numbers through electronics


Old school fractal printed out and taped to the side of a Vax. It is like chalking up a kill on the side of a WW2 fighter plane. Computing from the time where men were men and debuggers were nervous.

I cannot rationalise why these analog computers are so cool (pics here). I think its to do with getting to the bare metal of a computation. Also something of the history of these machines is amazing. Looking at the circuit boards you can imagine national characteristics they describe. The German machines based on mathematical abstractions are beautifully clean (and easy to fix apparently). The Japanese machines were a mess like their remote controls. The British machines had a distinct boffin feel to them. The machines from communist countries really seems like a scramble of clever but under resourced hacks. The American machines were big brash and heavy.


Kevin Kelly claims that a technology never dies

What happens is that most technologies become obsolete, diminish their role, but they don't disappear--more idea-based and can be resurrected more easily. Seems like a silly claim: the hand-axe, the arrowhead--they don't exist any more. And that's what I thought too... Looking on the Internet was able to find an example of whatever the challenger had given to me: brand new steam-powered valve for a steam-powered car; they're making flint axes exactly the same way, using exactly the same tools to the point they are almost indistinguishable from the original; archeological artifacts, making in huge numbers today.

But I really worry that the knowledge of many of the engineers who created these machines will be lost. One thing is that the analog museum really wants donations of machines that may be thrown out. If you see a box of the brand Dornier, TeleFunken, Gould, Aritma, BBC, Electronic Associates Inc, Solartron or even just something you think might be of interest send him an email. Even parts of these machines could be really useful to help repair the computers he has.

I do not think these machines are purely of historic interest. It is possible that in the future analog computers will be used in the real time simulations they excel in. One area I think they could be useful for is search engines. Linear algebra calculations are at the heart of modern search engines. this is described in
The $25,000,000,000∗ Eigenvector the linear algebra behind google and How Google Finds Your Needle in the Web's Haystack

These calculations do not need to be precise (the google algorithm uses random walks) which removes one of the big objections to analog computers.

George Dyson in the this years edge question 'What Scientific concept would improve everybody's cognitive toolkit?' describes a search engine based on analog computers here

Analog computing, once believed to be as extinct as the differential analyzer, has returned. Not for performing arithmetic — a task at which even a pocket calculator outperforms an analog computer — but for problems at which analog computing can do a better job not only of computing the answer, but of asking the questions and communicating the results. Who is friends with whom? For a small high school, you could construct a database to keep track of this, and update it every night to keep track of changes to the lists. If you want to answer this question, updated in real time, for 500 million people, your only hope is to build an analog computer. Sure, you may use digital components, but at a certain point the analog computing being performed by the system far exceeds the complexity of the digital code with which it is built. That's the genius that powers Facebook and its ilk. Your model of the social graph becomes the social graph, and updates itself.

Analog computer methods of calculating eigenvectors have been described decades ago (pdf here).

It is possible that the techniques of analog computers could be the future of search engines. We should make some effort to save these machines, any manuals written on them and if possible some of the knowledge of those who made and used them.

Think Different

I am fascinated by how people think differently from each other. I think there are big differences in the fundamental methods people use to think, but because we rarely talk about how we think these are not well known.

Feynman talks here about how he heard numbers in his head when he counted and his friend saw numbers visually.


He believed that such fundamental differences explained why we sometimes have such difficulties communicating with each other.

I 'see' numbers as a kind of line and I have heard other people describe other ways they 'see' numbers

Reading is another place people think differently. When I read I 'hear' the words as I read them. This is called subvocalisation and is normal but it slows down reading. "Subvocalization, or silent speech, is defined as the internal speech made when reading a word, thus allowing the reader to imagine the sound of the word as it is read". There are all sorts of online guides as to how to stop subvocalising. My wife does not subvocalise and never has. She describes seeing words as the actual word rather then hearing it and then understanding what it means.

Do left handed people who do that hook handed over the top writing have a fundamentally different view of writing? Do you know of any other basic cognitive tasks that people carry out in completely different ways?

NOW Who is the popular one?

Someone of the two of you readers were kind enough to nominate this blog for the Irish blog awards


Science/Education 2011 Best Science/Education Blog
Technology 2011 – Best Technology Blog/Blogger

Thank you very much to all of you who read my nonsense especially those who decided it was worth pointing out to other people

On a related trumpet blowing note Tyler Cowen thinks this blog shows that Ireland is going to collapse into anarchy,

@tylercowen
Will the Irish be more or less impatient than the Egyptians?, http://tinyurl.com/4zyyxsg

Which now I think about it may not be a complement

Congratulations to my friends in thebeernut, L. Mulligan grocers and Dungarvan Brewing Company on their nominations.

A no lose lottery would help Haiti

In this post I want to try and persuade you that there is a form of lottery the Prize Linked Savings scheme that would be popular and fairly cheap to run for the people in Haiti. In my next post I want to explain why such a scheme would actually be useful

"PLS is a kind of savings account that pools some of the interest from all depositors and pays out a big lottery prize every month or so. It combines the thrill of the lottery with the safety of a savings account. It’s sometimes called a “no-lose lottery,” since a depositor is automatically entered into the lottery but can’t lose the original money she deposits."

Lotteries are incredibly popular in Haiti. Probably due to the crushing poverty there the hope and 'chance to dream' lotteries provide is seen as vitally important

"the lottery system in which an astonishing number of Haitians invest their income and their dreams." "At the time of the last official count, in 2005, the streets of Port-au-Prince were home to nearly 2,000 of them" (borlettes lottery shops) "– more than double the number of schools and universities in the city"."All told, Haitians spend as much as $1.5 billion per year on the borlette – a staggering amount in a country whose gross domestic product last year was $6.9 billion"

"You can bet two balls to come up at once, and if that happens you make a ‘marriage’ and win 500 times your bet. If your married numbers are consecutive, you win 1,000 times your bet. The Haitian dream is a bet of US $1,000 that reaps US $10,000 on a consecutive marriage. Many a hopeful future has been built on less." Borlette lottery in Haiti is remarkably popular which is a good indication that a lottery that also acts as a savings scheme will be popular

The overhead cost of these Haitian lotteries is relatively low as they use the numbers and thus the security mechanisms of outside lotteries. These lotteries tend to use the numbers from other countries lotteries for example the New York State Lottery or the Santo Domingo State Lottery

Another expense is the prize won in the lottery. You want your prize to be equivalent to a million euro. This is an amazing life changing amount in Ireland where the average wage is 35K. So a million is about 30 times average income. For Haiti this would be $730 *30= 21 thousand dollars. A million Haitian gourde is about 25 thousand dollars.

If we had a monthly prize of this much that would cost 12*25k=300 thousand a year in prizes. Say you double that with some smaller prizes. 600 thousand which if you were earning 3% interest means you would have to take in 10,000000 dollars. This is a lot of money but in the context of $1.5 billion spent on lotteries in Haiti it is not that much.

If you could take in ten million dollars you could run a no lose lottery in Haiti and use the interest to pay for the prizes. But why would you want to? I will explain in the next post why PLS would be useful to the people of Haiti

The Great Stagnation in Ireland

There is a new ebook by Tyler Cowen about how growth rates and progress in general has declined since 1973. There is a podcast here where he discusses the argument.

The book, is short you can read it in an evening, and published only in ebook format. It is a bit like an evening with your smart friend as he explains something rather than an economic treatise thats full of equations and aids insomnia. It is well worth the 4 dollars and few hours it takes to read.

Loads of commentators have examined various parts of the argument for stagnating progress. I even joined the argument here. I am going to try give a few datapoints that relate to Ireland that are part of the great stagnation argument here. Ireland has not been stagnating since 1973. In that time we moved from an improvished bachwards country to a modern well off one (and then slipped back somewhat again).
But there some areas where we are stagnating in similar ways, there are areas where we made similar mistakes to the Americans (and the rest of the OECD) and if we can expect a similar decline in growth here that means we should alter our expectations.

Stagnating medicine

We spend too much on medicine and do not get good value for that spending. There is a piece here by Ronan Lyons where he argues convincingly that with our age profile we should be spending a third less than what we currently do on medicine


It is generally agreed that access to the health system in Ireland is bad. If we spend more than we should and people cannot get healthcare as easily as other OECD countries do it is fair to say that Irish healthcare is not improving our quality of life the way it should be

Stagnating education

'In the past decade, funding per student has been increased in real terms by 61 per cent, yet performance has decreased by 15 per cent.

This disconcerting news was reinforced before Christmas when the international ranking of the reading, mathematics and science skills of Ireland’s 15-year-olds was published. Reading skills dropped from fifth to 17th, the sharpest drop among 39 countries, with one quarter of all 15-year-olds classified as effectively illiterate. This was more politely termed in the report as “below the level of literacy needed to participate effectively in society”.'

We are spending more and getting less on education. Literacy is one of those basic measures that is hard to fake. Anyone who leaves school illiterate is a failure of our education system. In this school education area Ireland seems to have stagnating improvement.

Expecting too high growth
One of the major conclusions of the book is that Americans expected growth to continue at previous rates and this facilitated the financial crisis. People believed house prices would go up forever without wages increasing at the same rate. When people realised that they could not afford their debts and no one was going to buy them off them the house of cards collapsed.

In Michael Lewis' Vanity Fair piece makes a similar claim that Ireland expected too much growth
'An Irish businessman named Denis O’Brien sat on the board of the Bank of Ireland in 2005, when it was faced with the astonishing growth of Anglo Irish, which was about to double in size in just two years. “I remember the C.E.O. coming in and saying, ‘We’re going to grow at 30 percent a year,’” O’Brien tells me. “I said, How the fuck are you going to do that? Banking is a 5-to-7-percent-a-year growth business at best.”'

If the financial crisis in America was aided by delusions of future growth the Irish collapse was to an even greater extent.

Expecting growth to be much higher than it will be leads you to paying too much for things. For example buying empty buildings on the grounds that NAMA will make a profit on them eventually.

Other Claims
There is a very interesting section on how the internet might entertain us but does not provide many jobs. There is a similar argument to in "An Exodus Recession?" that because we can sit at home and look at youtube essentially for free the recession both does not feel as bad but is measurably worse as people do not go out and spend money on cinemas etc that employ people.
This does not relate specifically to Ireland though I do think the internet has improved our lives in ways that are hard to measure and that probably help those who like reading and arguing more than they help the average person.

Cowen also claims that increased emphasis on scientific research could again get progress moving faster again. I was watching Bronowski's the ascent of man last night from 1973, the year Cowen claims the stagnation began. Bronowski gives a warning that the next step in mankinds progress is dependent on science and that if we don't prevent scientific progress from stagnation some other civilisation will.


Expecting growth to return now

I have heard many people running for office who claim anyone who questions that we can return to the good old days of 5%+ growth of being unpatriotic. In this narrative our debts are huge but they were in the 1980's as well ans well soon have 5%+ growth that will evaporate our debts for us.

The evidence is that once countries get as rich as we are now their quick growth from playing catchup then slows. Some of us might be able to get more productive now but not nearly the increase that happened in the 90's when we went from working on farms producing vegtables and potatoes to producing Viagra and Pentiums.

We are not going to have the growth rates we had in the 90's we like the rest of the OECD have entered a period of lower growth, what Tyler Cowen calls 'the great stagnation'. Anyone who says we should plan for Ireland growing at a rate higher then the 6.7% interest rate on the ECB/IMF loan are is trying to get you to take on billions of their debt.

The great stagnation a great short book. Though its arguments apply to America the lessons of what they Japan and other countries who developed before Ireland are important for us to learn.

Watson is better off having a three way competition

It seems non intuitive but I think Watson is more likely to win because its competition are so good. If you look at how well it did on Jeopardy here it seems to show Watson is better having humans splitting the points on questions it cannot answer as it is so quick to answer the ones it can.

When the humans buzz in they are right 19/23 about 83% of the time. Suppose Jennings was up against watson and me and I got zero buzz ins. Jennings may have gotten 11 of the buzz ins that Rutter got. Watson seems so fast on the buzzer that it seems to buzz first for nearly every question it could answer 44/50 times it tries to answer it is first.

This means that Jennings could nearly have doubled the number of answers he had but because there are two people who are really good at the sorts of questions only humans could answer these points are divided amongst the humans. Of the 23 human answers only 6 did Watson try to buzz in on.

There is a similar probability puzzle that might help in a three way duel. "You're a cowboy, and get involved in a three way pistol duel with two other cowboys. You are a poor shot, with an accuracy of only 33%. The other two cowboys shoot with accuracies of 50% and 100%, respectively. The rules of the duel are one shot per cowboy per round. The shooting order is from worst shooter to best shooter, so you get to shoot first, the 50% guy goes second, and the 100% guy goes third, then repeat. If a cowboy is shot he's out for good, and his turn is skipped. Where or who should you shoot first?"

Theres another explanation of a three way duel here.

Sometimes having 2 really good competitors increases your chances of winning.