Not quite, but these are awesome!

Each card has a cute picture on the front and some key takeaway messages on the back. The idea is to give these kinds of cards out at conferences or events so that people who attend might actually remember something from a presentation and look into it more later.

I realize I missed the boat on these – they have been around for a while. But getting to handle some today just felt like so much fun! I would almost like to suggest that every academic paper should have one. The cards gloss over the fine details, but most people wouldn’t read the fine details without a nice card luring them in. It also would help the researcher. If you have to explain your paper in 3-5 bullet points, you will really have to focus on the important parts and communicate better.

How to know you have hit upon a very controversial subject: two titans of development economics each castigate you for diametrically opposite reasons. Next time, I should let them fight directly!

I am trying to use AidGrade’s data to say something about the generalizability of impact evaluation results. I’m not coming in with an agenda, but basing this on the belief that:

1) People want to know what works. There are a lot of grandiose claims that impact evaluation can tell us this. Economists are usually very careful not to generalize from particular cases, knowing that results are heavily context-dependent and have no external validity. But there is also a sense in which we really do want to use the results to update our priors. We want to get something generalizable out of an impact evaluation, else why do one in the first place if it only tells us how successful something which will never again occur was?

The extent to which results are generalizable is an empirical question. So long as people are extrapolating from past results, whether explicitly or with a wink and a nudge when trying to get policy makers to agree to a new impact evaluation, we’d better at least know how generalizable the results are. You can say we know they aren’t generalizable. Fine. People still talk as though they are, they are likely to be to some non-zero degree, so what is that degree?

2) There are undoubtedly contexts under which results are more or less generalizable in practice. For example, since a lot of people don’t want to randomize, I suspect that RCTs may be done in “weirder” situations than quasi-experimental studies. I wonder if we can see this in the results. Some causal chains between interventions and outcomes may also be more complicated than others. The theory here is quite clear – why not test it out?

Unfortunately, I’m stuck between a rock and a hard place. Some say it goes too far, others not far enough. I’m a fan of impact evaluations for what they do tell us about human behaviour. I also think they are often vastly overpriced and that many but not all of them would be more helpful were they to give more immediate, actionable feedback to the project implementer.

I’m not actually interested in participating in the War of the Randomistas. But when a war goes on, it seems that some on either side have hammers and everything looks like a nail.

If special interests kill it, so be it, but practically speaking it’s an important issue.

Briefly: BITSS is highly necessary.

As mentioned in the new transparency series on CEGA’s blog, there are a lot of problems in the discipline today. Focusing on interaction terms or particular subgroups is one way of increasing the odds of obtaining the elusive 5% significance level, so a lot of people do it, but this overstates the results’ true significance. Something can always appear significant by accident, and tests of significance are only legitimate if they are defined beforehand.

Because of this issue, AidGrade has been forced to take a very conservative stance with regards to the values it collects from studies. In the absence of pre-analysis plans, we have avoided collecting interaction terms and have also focused on results containing few controls (another way that people can lie with statistics). This is obviously a shame, because there is a real sense in which these terms could be important. However, in the absence of pre-analysis plans, this is the best we can do in order to avoid fraud.

Here’s hoping the discipline will change with all the new attention brought to bear on this topic!

AidGrade finished its topic selection for the next round of meta-analyses.

One thing that we’re doing in this round is to randomize which topics we are working on (apart from also taking the winner of the popular vote).

Suffice it to say, I’m sure none of us is perfectly happy with the topics we are pursuing, as we all have our favourites. Not being able to choose personal favourites is a real drag, so I can well understand the pain of those who don’t want to randomize in their own projects. That said, I am very excited about the venture and it is developing very nicely!

P.S. You can read more about our process here.

After realizing that I save a few thousand dollars per year buying in bulk off the internet, I started to wonder how the CPI takes online shopping into account.

The good news: it does try to. It has a telephone survey (TPOPS) that it uses to try to re-weight where people are shopping, including online. Unfortunately, I couldn’t find any actual copies of this survey online, and the best information I could find on how the BLS tries to deal with this issue in general is here.

Independently, it seems like the CPI will get wackier and wackier. As this article points out, the internet lets people buy pretty weird stuff. How will the BLS capture that diversity while keeping surveys short? They say TPOPS should only take 7-15 minutes.

Again, I wish I could find a copy online, but if they were to ask me about specific food items, for example, they probably wouldn’t even think to ask about half the food I eat. And we are all getting weirder and weirder about the food we eat! If they were to instead ask me about food as a general category, I probably would forget about my online purchases even though I get all my staples there, because I would associate food shopping with buying fresh fruit and vegetables from a brick-and-mortar store.

As a whole, online shopping still isn’t a big deal to most of America. But it’s getting bigger. Expect these issues to become more and more important.

This is a very old idea that I’m putting out in the public domain so that someone can use it. I’ve already shared it with several academics but so far nobody has tried to use it to my knowledge….

For dual-academic couples: one person publishes an academic article and puts in the acknowledgements “X, will you marry me?” The other person then has to either publish another article with their reply or perhaps write up their reply as a comment to the first article!

Reminded of it as I see this is making the rounds. Yes, I know, my idea takes a lot longer!

Over 100 million impressions. 50,000 tweets. Twitter dump here.

This might be the last update of the twitter dump, as others have continued to scrape the tweets and have added in some of the data from the older tweets. Please let me know if you’d like the occasional update to continue here in .csv form (e.g. for some of the other data contained in the tweets beyond the scraped pdfs).

Pros:
- You get to see it mentioned on almost every major online news source (NYTimes, FT, BBC, Washington Post, CNN, NBC, CBC, Al Jazeera, Bloomberg, Businessweek, FastCompany, Huffington Post, SFGate, The Daily Beast, Slate…), almost every major tech source (TechCrunch, Gizmodo, Arstechnica, CNET, PCMagazine…) and some cool ones like Nature’s blog, the Chronicle of Higher Education and Wikipedia!
- You meet a great group of people.
- Long-lost friends and acquaintances get in touch.
- You get a better understanding of how journalism works.

Cons:
- You get a better understanding of how journalism works (just joking!).
- It really screws with the results that appear if someone googles you. I have done more things than coin a hashtag, people.
- You get almost no sleep for several days.

I alluded to this before: I have a lot of things depending on me that are not this. I’m not the one to carry on the fight, though I may post from time to time about it.

Finally: tweeting can’t be the end, but it’s a very good start. Apart from the arguments previously posted, there’s a behavioural economics literature that says that when people commit to something publicly, they are more likely to do it. I hope the tens of thousands of people who tweeted about #pdftribute continue to feel a connection to the movement, keep sharing their papers and keep pressing the system to change.

(NB: Since there might also be an “I already did my part” effect, the effects of tweeting could in principle go either way, but that’s my bet.)

#pdftribute as a hashtag is dying down, but several concrete endeavours are arising from many good people who were involved in or motivated by it.

Here are a couple of initiatives about which I am aware (merely curating; this does not count as an endorsement):

1) The Papester Collective. Need to get behind a paywall? Send a tweet.

2) Github for research. Searchable paper repository. Easy upload, perhaps with a tweet. Brought to you by a collaboration of the people behind pdftribute.net, http://edward.io/pdftribute, @tmccormick, @thejbf, and @mrgunn at Mendeley.

Some people have argued that since these kinds of ventures don’t change the underlying incentives for academics to publish in top journals, they won’t change the system. I beg to differ. You don’t need to change the academics’ incentives to publish in top journals. You need to change their incentives to share their papers regardless or to have others share their papers regardless. For example, it is usually perfectly fine to share late stage working papers, which many already do to help others gain access to their findings without being stuck at a paywall. These kinds of sharing mechanisms could in principle work, just as Napster changed the music industry.

Here are some other ideas from the comments received on twitter. What are your comments?

@evavivalt love to see people discover if their institution or subject area has a repository where their work can be deposited #pdftribute

— Ed Summers (@edsu) January 16, 2013

@thatandromeda @evavivalt What if libraries built + supported open publishing in a big way, cutting the middle man making scholarship open.

— Gabe Gossett (@Gabe_Gossett) January 16, 2013

@evavivalt @venturejessica @mattyglesias Now we must demand #JSTOR be #openaccess. I can’t use JSTOR again w/out thinking of a child dying.

— DCProfessor (@DCProfessor1) January 15, 2013

I want it to be clear that pdftribute is not “my” movement. It isn’t really anyone’s movement per se – if it were anyone’s, it probably wouldn’t be a movement!

My involvement has been: thinking to put out the initial tweet, publicizing it, creating the hashtag #pdftribute, based on a suggestion by @venturejessica, collecting the stream of tweets and sharing it for others to scrape, and tweeting like hell for the past 48 hours or whatever (despite unreliable wifi).

Everyone else has also put in a lot. @venturejessica came up with #pdftributetoaaronswartz and got interest from Anonymous, which really helped get this off the ground. Micah Allen had suggested something similar on reddit, which might have helped it be picked up later. Thousands of people tweeting on the internet contributed as much as anyone else. The pdftribute.net guys have really been great – and now Edward Wang and James Change answered the call to scrape the pdfs from the tweets and have made them available here.

1) The kid in me has to say: SO COOL.

2) I didn’t think, when making my pledge, that it would develop into anything this big. It just seemed like a no-brainer; the completely obvious thing to do. It’s not clear yet how things will develop from here, but now that so many people are taking part and carrying this on in so many ways (and there is plenty of chatter – let me know if you would like to join the conversation), I will happily recuse myself to focus on my other endeavours. Movements like this don’t need a founder. They outlast any of us.