Two weeks ago Mike Lilley of the Sunlight Policy Center of New Jersey skewered the scholarship of Mark Weber, aka Jersey Jazzman, in an analysis called “More Shoddy Research from Jersey Jazzman and New Jersey Policy Perspective.” In Weber’s paper,“New Jersey’s Shrinking Pool of Teacher Candidates,” Lilley shows how Weber cherry-picks data, uses unsupported premises, and confuses correlation with causation. For example, in order to show a “wage gap” between public school teachers and credentialed professionals, Weber includes the lower wages of private school teachers. In order to buck up a fallacious claim that NJ teachers suffer in comparison to private employees, Weber understates the value of NJ teachers’ pension benefits. (I summarized the dispute here.)

Last week Weber published a rebuttal of Lilley’s analysis on his blog. Yesterday Lilley posted his reply, which I’ve copied below.

First, a little twitter activity:

Keith Benson, head of the Camden Education Association who insists that the terrorist attacks on September 11th were a conspiracy of the American government, called Lilley a “shill” who issues “high-priced crap.”

The sane blogger at New Jersey Education Aid and Lilley had this exchange:

Here is Lilley’s response to Weber’s rebuttal of Lilley’s rebuttal of Weber’s paper.

Sunlight (not “Sunshine,” Jazzman) Policy Center of New Jersey (SPCNJ) will give credit where credit is due: at least The Jazzman (a.k.a., Mark Weber, Ph.D.) took the time to defend his research.  Presumably, no self-respecting Ph.D. would allow SPCNJ’s critique “More Shoddy Research from Jersey Jazzman and New Jersey Policy Perspective” to go unchallenged.

However, Weber’s defense is as flawed as his original work.

1. Weber attempts to dismiss SPCNJ’s criticism of his cherry-picking data as “silly nit-picking.”  The very first data source Weber cites on a teacher pay-gap is the EPI study by Allegretto and Mishel, citing a record-high national pay-gap of 21.4% – an eye-catching number, to be sure.  Nowhere in the main body of his report does Weber acknowledge that the same study calculated New Jersey’s pay-gap as well.  In a study about a pay-gap in NJ, why wouldn’t Weber cite the NJ-specific data in his main body?  Because it undermines his entire case.  Only on the very last page of the Appendix do we learn that Allegretto and Mishel found only a 3.8% pay-gap in NJ.  This is the very definition of “burying” data.  Weber unconvincingly contends that this is an “outlier” due to changes in methodology, but if their national data is worthy of prominent citation, why isn’t their NJ-specific data?  Bottom line:  Weber buried this relevant, NJ-specific data on the last page of the Appendix.  That’s called “cherry-picking” your data, which has no place in legitimate research.  
2. Weber states that he uses IPUMS data for his pay-gap calculations, and he acknowledges that because this data includes private-school teachers, this likely increases the pay-gap.  Weber states that the IPUMS study is a survey and speculates that there are all sorts of reasons why it might register lower teacher salaries.  The NJDOE data that Weber does not use in his analysis (but does mention in the Appendix) is for public school teachers only and reflects what they were actually paid.  Why wouldn’t Weber use this data when analyzing a pay-gap for NJ public school teachers?  Because, as we learn in the Appendix but not the main body, it shows that actual teacher salaries are 3.6 to 7.8% higher than the IPUMS survey data, and therefore narrows the pay-gap.   Bottom line: Once again, when the data is inconvenient, it’s not presented in the main body but is relegated to the Appendix.  More cherry-picking.  
3. Weber claims that SPCNJ is trying to “prove a teacher wage advantage.”  This is false.  SPCNJ attempted to prove no such thing.  We made clear that there may be a case to be made that there is a pay-gap but that Weber did not make it.  As the title of the piece said: “Shoddy Research Does Not Help the Cause of New Jersey Teachers.”  Bottom line: SPCNJ showed that when scrutinized closely, the very same sources that Weber cited provided data that undermined his entire case.   No wonder Weber would rather call SPCNJ names than deal with this foundational flaw in his research.
4. Weber tries to dodge the issue of valuing pension and health benefits when looking at overall teacher compensation.  Weber would rather use teacher salary data only and leave benefits out of it.  Why?  Because it is widely known that public teachers in NJ receive much more generous benefits than private-sector workers.  Weber conveniently makes no attempt to determine the value these benefits for NJ teachers and simply concludes that “there is little reason to believe non-wage compensation makes up for New Jersey’s teacher pay gap.”  Bottom line: This is not research, this is supposition.  He does however cite to comparisons between NJ’s public pensions and other states’ public pensions.  SPCNJ said it once and we will say it again: such comparisons are totally irrelevant to a study of a NJ public teachers – private-sector pay gap.
5. Weber simply ignores SPCNJ’s data that shows it is invalid for Weber to assume that all college degrees have the same economic value.  Why?  Because if all college degrees are valued the same, then there is indeed a pay-gap between teachers and private-sector workers.  But all college degrees are not the same: engineering and computer science degrees have greater economic value than an education degree.  Bottom line: Weber’s assumption is simply invalid.  
6. Weber claims that SPCNJ agrees with him about the primary conclusion of his second report: the reality of a decline in the number of teacher candidates.  But this is not accurate.  Weber’s primary conclusion dealt with the causes of the decline.  As stated in his last paragraph and in his “Conclusions and Recommendations,” the state should “act immediately to make the teaching profession more attractive, including raining pay, shoring up benefits, and increasing the respect” for teachers.  Bottom line: SPCNJ’s criticism is that Weber offers no proof for his conclusion as to the causes.  He presents a lot of unproven assertions and suppositions but no actual research to support his conclusion.  In his response blog, Weber claims that “Research shows teacher compensation affects decisions to enter the profession.”  If so, where is that research?  And why didn’t he include it in his research report in the first place?  Weber, himself, has implicitly acknowledged that there was a substantial gap in his research.   There were many others as well.  Please see SPCNJ’s recent report for a full accounting of the many flaws.
7. SPCNJ focused on the errors in the parts of Weber’s reports that we have institutional knowledge of.  SPCNJ does not believe it is competent to critique the other areas Weber studied.  That is not to say they aren’t important.  It is to say that SPCNJ tries to focus on issues it knows and present facts and data that are sourced and footnoted rather than mere opinions or conjectures.  SPCNJ believes this is an attribute, not a fault.

THE BOTTOM LINE IS THAT WEBER’S DEFENSE IS AS FLAWED AS HIS ORIGINAL WORK.  HIS TWO RESEARCH REPORTS FOR NEW JERSEY POLICY PERSPECTIVE STAND AS NON-EVIDENCED-BASED AND SUBSTANDARD.