Proverbs

1.  It's not about Big Data, it's about the right data!
2. Data without theory is meaningless; theory without data is sterile.
3. A peer-reviewed publication isn't a proof. At best it's a line in a proof.

Socially Integrated Computing: A Vision for Computer Science

What is Computer Science?

Computer science as a field started in the early 20th century with the first general purpose electronic computers, and the theoretical models of Turing, Church and others. As ideas of computational organization and architecture were developed and notions of (hardware and software) layers of abstraction evolved, the field burgeoned and grew in many directions, spawning many subfields (architecture, programming languages, algorithmic design, complexity theory, and so on). Overall, though, the fundamental object of study in the field is the computational system, which can be thought of as a kind of machine inside a box — we feed inputs into the box, and receive outputs from the box (computed by the machine).

The evolution of the field can be understood as the progressively deeper understanding of, and the ability to engineer, such computational systems. This can be best understood as growth along three interrelated dimensions: power, trustworthiness, and reach. 

Power is the ability to perform larger and more difficult computations. Faster and larger computing machines increase power, as does the development of faster algorithms. Complexity bounds enable us to better understand the limits of computational power, and programming language structures enable effective construction of more powerful software systems.

Trustworthiness refers to what extent we can ensure that our systems are reliable and trusted by users at the task(s) they are designed and deployed to perform. Cybersecurity measures work to ensure trustworthiness in the face of external threats; program verification and software engineering techniques to ensure it against (inevitable) human error. Consideration of human factors is also essential for computational systems to be trusted by users who are not privy to the system’s internal workings or design process.

Reach is about how widely computational systems touch and influence human activities. The earliest electronic computers had limited reach, serving a limited number of needs of government and large businesses, while today we carry powerful hand-held computers serving a great variety of personal and social needs, and the growth of IoT promises to increase computing’s reach even more.

Growth in each of these dimensions requires, influences, and constraints growth in the others. Increasing power or reach opens up new gaps in trustworthiness that need to be addressed, while increasing trustworthiness and reach tends to require more computational power.

Current Trends

We can understand the implications of current trends in computing via this tripartite lens, as well as the relationships between these developments.

• Recent successes in machine learning with big data have increased the reach of computing systems into new segments of society and the economy, therefore also highlighting new aspects of trust that need to be addressed, in achieving fairness and dealing with bias in analytical and decision support systems.
• Edge computing is about increasing computational power by redistribution of computation, concomitantly enabling the increase of reach, thus also requiring deeper attention to questions of trust in cybersecurity.
• The complementary tech areas of Internet of things and intelligent spaces also increase reach greatly into our lives and daily activities, raising questions of trust related both to cybersecurity and to how understandable and predictable their responses are to us.
• Similarly, digital manufacturing increases the reach of computing into the physical realm and gives us new questions of trust in cybersecurity and complex logistics management system that need to be solved, as does digital twinning. 
• Development of more realistic virtual and augmented reality systems promise a radical increase in computing’s reach, and profound social impact. This has been enabled by increases in computing power, and to reach full realism (however defined) will require even more power (cf. edge computing). It will also raise new questions of trust, particularly around privacy, as well as the effects of trust breaches when computing is tightly integrated with everyday activities
• Quantum computing promises greater power, and specifically power which undermines public-key cryptography, a main pillar of current computational trustworthiness.
• On the other hand, blockchain and other distributed ledger enable the creation of distributed systems that can be trusted, even if individual actors using the system are not (zero-trust information security), and that are robust to losses on the network (due to distribution). This uses the power of the network to create systems that increase trust and thus enables increased computational reach (via new financial and other applications).

The Next Step

Where, then, is the edge of the field where innovative research will move the field forward? 

At the macro level, we see a great deal of possibility in all three main themes: Computational power has reached the limits of Moore’s Law, and so is ripe for a deep conceptual shift, whether through quantum, parallelism, non-von architectures, or some idea yet to be discovered; Computational trust is strained (at best) by increases in power and reach of computing - cybersecurity is an eternal arms-race, catalyzed further by demands for efficiency, and increased use of automation (whether AI or otherwise) raises new and evolving questions of algorithmic trust; Computational reach has increased enormously through the ubiquity of computing devices, both personal and IoT, which spread computation to new aspects of our societies, economies, and lives.

In fact, if we consider the effects of many of the current trends identified above we see one theme emerging overall — the growing intertwining of computational systems with human systems (individual, organizational, social). Personal devices (phones, watches, etc.), IoT, and intelligent spaces deeply connect our daily and minute-to-minute activities with computational adjuncts; as virtual/augmented reality becomes good enough for broad use, this integration will leap even further. In industry, digital twinning and digital manufacturing put computational models at the center of physical industry. And, while use of information technology has a long history in finance, the way blockchain has enabled creation of non-governmental currencies, and how modern data analytics has increased the power and reach of high-frequency trading and development of complex financial instruments, has transformed the industry (and left regulation somewhat behind for now). Similarly, e-discovery is transforming staid law offices, computer vision food safety as well as law enforcement, and machine learning systems are being applied (rightly or wrongly) to a whole host of societal, political, and business problems.

Thus, the broad paradigm shift that is needed is a change in how we think about the object that we study in computer science. Rather than it being a metaphorical glass box containing just a computational mechanism, receiving inputs and providing outputs to an external user

we need to expand the box we consider to include also the user and their behavior, as part of a larger, more complex computational system

and even beyond the user, also their social/organizational context 

This system has enormously more degrees of freedom, and is not fully controllable by the computer scientist, but cannot be ignored, due to the tight interconnections between the core computational system (hardware, algorithm, data structures) and the information flow and incentives induced in its human users. This view of the proper object of study of our field we may term socially integrated computing, in that the human and social context (understood broadly to include all relevant human-human connections and interactions) are taken as integral to the computational system to be analyzed and designed.

There is, of course, much work in this vein already, in HCI, in social network analysis, in agent-based modeling, in computational economics and mechanism design, and so on. I believe, though, that these disparate types of research work can best be understood together as aspects of a shift in how we view the field as a whole — socially integrated computing — and that such a unifying view ought to transform it, and its effect on the world, for good.

Commencement 2020

It is a great shame that universities were unable to hold a regular commencement this year, and honor the achievement of our graduates, not only to complete a university degree, but to do so under nearly-unprecedented conditions, with their entire educational system up-ended in the middle of their final semester.

The following is the text of my speech to the graduates of the Illinois Tech Computer Science Department this year:

It is my great pleasure to welcome you all today. I am Shlomo Engelson Argamon, Interim Chair of the Computer Science Department at the Illinois Institute of Technology. 

None of us ever expected that your graduation would be like this—we should be together on campus, celebrating you and your accomplishments as you all truly deserve. I look forward to the day that we can invite you to return to campus for our next commencement, and I hope many of you can, so that I can congratulate you in person. 

Today is a day of celebration as we gather virtually to recognize your accomplishments and honor each of you as a 2020 Illinois Tech graduate of the Computer Science Department. I only wish we could now be together, with your parents, family, and friends, to share this great moment of your graduation. 

Congratulations! Your hard work at Illinois Tech has set the foundation for your future professional success. 

You are graduating into a world of enormous challenges – the reverberations from this pandemic will echo economically and socially for many years to come. And for you--your final year at Illinois Tech has sadly been disrupted, and you may soon or already be facing personal challenges as well. If you do, I pray you overcome them with grace and dignity.

Amidst all these challenges, you are also graduating Illinois Tech into the field that will be central to addressing them – indeed, without your field, we could not even have this event today. Ironically, computer scientists, the stereotypical socially-inept geeks, are the ones now enabling people to remain connected with one another in these difficult times.  

Computer science is not and can not just be pure technical brilliance walled off from the messiness of human relationships and interactions. The systems we create now mediate and shape human society in profound ways, both beneficial and harmful. 

Many of you, I expect, will become great innovators, as researchers, developers, entrepreneurs. I urge you to not get lost in the admittedly fascinating technical complexities of our discipline. Always consider carefully the larger “computational system” of people, and relationships, and organizations within which your work is embedded.

This is both a technical, and a moral, imperative.

Disinformation now is rampant, spread and incentivized as a side-effect of the algorithms that make social media efficiently monetizable—we are slowly starting to realize how self-driving car modes encourage reckless driver behavior—holes in system security inevitably lead to terrorizing children through internet-enabled baby monitors, or videoconferences through zoombombing, all for the lulz—and on and on.

The future of our field and its enormous effects on the world depend crucially on this fundamentally human perspective, which will affect not only the applications we develop but the very nature of the field itself. 

Illinois Tech was founded 130 years ago on the idea that a first-rate technical education could enable individuals to change their lives and the world for the better. We have endeavored to give you that education, the rest is up to you.

Graduates of 2020 – I cannot wait to see what wonders you each create. Dream high, and celebrate your successes, both large and small.

As of today, you are all Illinois Tech alumni. Bear that title proudly as you move on in your lives, and please stay in touch with the Computer Science Department, with Illinois Tech, and with one another. Wherever you may go, we at Illinois Tech will follow your future accomplishments with keen enthusiasm.

Musing: Role of Attention in Language Development

Some thoughts about developmental linguistics, based on observations of my kids. These ideas are admittedly naïve, and I don't know the relevant literature, so would appreciate any expert feedback, whether support or refutation, if these ideas are old hat.

Observation 1.

My 8-year-old frequently says things like, "John, he was playing basketball," topicalizing the subject by left-dislocating it. (I'm not sure if this is only for animates.) This is not usual in our dialect, so I doubt that the kid has often heard it. My wife and I incessantly correct this usage (yes, yes, intellectually I am a good descriptivist, practically, however...), and yet it persists strongly.

This is rather interesting, I think.

To the best of my (limited) understanding, most syntactic formalisms are structured around subject-predicate relations, and treat information structure as piggy-backing on fundamental mechanisms, but this observation makes me wonder if information structure might be more fundamental in some sense. 

Observation 2.

Observation of my (slightly language delayed) not-yet-two-year-old leads me to consider further a foundational role for attention and indexicals. (I know the latter is a rather old idea.)  

The child has excellent comprehension, as far as we can tell, as he responds appropriately to quite complex utterances, and makes himself understood (with difficulty) mostly through grunts and gestures (and a slowly growing vocabulary).  

What I've noticed, though, is that he achieves communication largely by coordinating and controlling mutual attention. He will gesture so to capture my attention and direct it to an important object, and in fact will often grab my face and turn it to face him as he gestures, or to face what he is interested in, before gesturing and grunting his "utterance".  

Much of his speech therapy also uses attention as a key mechanism; vocabulary is given to him as he attends to certain objects in play, or to draw his attention to those objects. 

The primacy of vocatives in early speech is also closely related to attention coordination - very simply, a vocative is "getting someone's attention".

Information structure, whether topic, focus, or given/new is also about coordinating attention between the speaker and hearer.

Language development would seem, therefore, to depend critically on elements of theory-of-mind having to do with (at least) attentive focus (and related notions such as affordances and intentions). Notions such as "reference" would thus be constructed socially, in a sense, by mutual attentive connection of a linguistic unit and the referent.

(I think the notion of mutual attentive focus largely solves Quine's "gavagai" conundrum.)

Please forgive me if these musings are old-hat, utterly naïve, or just plain stupid. But if you are better informed about these matters, please comment with some relevant reading...

There is always bias (or, binary numbers are not the villain)

Twain Liu just wrote a piece on Quartz, entitled "Aristotle’s binary philosophies created today’s AI bias". This article is riddled with buzzword-laden arguments by feeling, such as summarizing "the very system on which all modern technology is built" as:

1 = true = rational = right = male
0 = false = emotional = left = female

The false dichotomies built into this asseveration rattle the brain. Indeed, the entire essay has this flavor, and it is not even false.

But since the piece has gotten a fair bit of attention, I feel the need to respond to the key claim of the piece. The entire argument rests on the dual assertion that the fact that computers use binary numbers (1s and 0s) as the basis for their operation is (a) based on Aristotle's (elitist, sexist) philosophy, and (b) the fundamental reason why algorithmic systems are biased. Hence, new computer systems not based on Aristotelian "binary" logic can be universal, unbiased pure goodness.

Well.

First off, the "computers are binary and essentially invented by Aristotle" claim is a load of argle-bargle and pure applesauce. (Clickbait headlines in the Atlantic notwithstanding.) When electronic computers were first being developed in the 40s and 50s, different systems were experimented with (including ternary (three-valued) logic), but binary was the most practical for a simple reason. With binary logic, you can represent a 0 by "voltage close to 0" and 1 by "voltage close to maximum". When you introduce more possible values, the system becomes more sensitive to noise, and hence less reliable. (There are other technical reasons for binary computing, and there are some other reasons to prefer ternary systems, but this is enough for my purposes.)

Now, to bias. Binary numbers have nothing whatsoever to do with algorithmic bias. The binary number system does not limit you to using only 1 or 0 for values you need to represent (after all, you could not specify an address to Google Maps just as a 1 or 0, say). Indeed, you can represent as many different values as you like by stringing bits together. You can have as many categories of whatever you like as you like. Any computer scientist would recognize this aspect of the claim to be laughable.

Algorithmic bias is due to the simple fact that all decision systems have biases. (Indeed, it is impossible to learn anything from experience without some sort of bias.) No real system has perfect information, and any decision made on the basis of imperfect information is biased in some way. The question is not "Can we create unbiased algorithms?" but "Do we know what our algorithm's biases are?" and "Can we mitigate the ones we do not like?"

Utopian visions like Ms. Liu's that if we just had the right philosophy, we could build computer systems that will be universal and unbiased, pure purveyors of algorithmic goodness, are false and actually dangerous. They promote the technocratic idea that there are unbiased algorithms out there, if we could just find them, and so keep our focus on algorithmic development.

However, bias is inevitable. The way to combat pernicious bias is through continuous monitoring to discover instances of problematic bias and exercise of good judgment to adjust systems (whether algorithms, training data, how the systems are used, etc.) to mitigate the bad effects while maintaining the good ones. The proper way to combat algorithmic bias (which some are working on) is to develop better ways of detecting and characterizing such bias, and the societal institutions and incentives that enable dealing with deleterious such biases. (And this leads into questions of value systems and politics, which cannot be avoided in this arena. There is no royal road.)

Visions of simple solutions derived from proper thinking are seductive. But the necessary condition for developing and maintaining diversity-enhancing technologies will be, I'm afraid, eternal vigilance.

Open Letter to PSU VP of Research Mark McLellan

Peter Boghossian is an assistant professor of philosophy at Portland State University. He recently, with two non-academic colleagues, published an account of an effort they made to probe peer-review methods within certain fields of inquiry that they term "grievance studies". Briefly, they wrote academic articles based on fanciful theories and hypotheses, matching as well as possible the style of writing and argumentation in the fields they addressed, and managed to get several articles accepted at leading journals. After doing so, they published their account of their effort, revealing the deception. This, they argue, has implications regarding the reliability of peer-review in those fields and perhaps regarding the legitimacy of the fields' methods themselves. I express no opinion regarding their study or conclusions.

What I am writing about is the response of Boghossian's institution, which was to investigate him for research impropriety, and ultimately to determine that Boghossian's "efforts to conduct human subjects research at PSU without a submitted nor approved protocol is a clear violation of the policies of [his] employer."

Unless the facts are substantially different from what has been published, this case raises concern about academic freedom and freedom of inquiry. It is debatable at best whether Boghossian's work required IRB review at all, and even if it had, the situation is not one to rise to the level of research malfeasance. If any readers have more information about the case, please let me know.

Below is the letter I wrote about the case this morning to Prof. Mark McLellan, Vice President for Research and Graduate Studies at Portland State University. Obviously, I speak only for myself, not for my institution.


Dear Prof. McLellan,

I have read with some concern of the investigation and conviction of Prof. Peter Boghossian of unethical research practices. This is a serious charge and as such warrants proper due process and full consideration of all relevant facts and circumstances. For the reasons I will detail below, I believe this not to have been the case here, and I urge reconsideration of this case, for the sake not only of Prof. Boghossian, but rather of the reputation of Portland State and the institution of the IRB.

I also note that, generally speaking, a first accusation or offense of this kind (lack of IRB review for research that did not result in proven tangible harm) will result in a warning and discussion with the faculty member, before proclaiming a determination that they have unambiguously violated ethical norms and university policy. Consider, for example, the very long time and repeated discoveries of egregious and intentional research malfeasance (far beyond anything that Prof. Boghossian is accused of) that were necessary before Dr. Wansink was finally censured by Cornell.

In the case of Prof. Boghossian, there are three essential questions whose answers would determine whether the project was subject to IRB review, and whether the project as conducted was unethical in any way.

First, was the project "research"? I believe the answer here is indeed "yes", since the project was undertaken to develop knowledge and disseminate it, in this case about the peer-review practices in certain fields of inquiry.

Second, did the project involve "human subjects"? Clearly, the fabricated research studies used as experimental probes did not. The reviewers of these articles, while part of the phenomenon under study (the "peer-review system") also were not human subjects per PSU's Human Subjects Research Review Committee Policy, which states:

A human subject is a living individual about whom an investigator obtains data, either from intervention or interaction with the individual, or through records which contain identifiable private information.

Since the peer reviewers were entirely anonymous and not identifiable, the investigators cannot be considered to have been obtaining data about them - no private information whatsoever was gathered, and the reviewers were performing their usual professional function. Thus they cannot have been considered human subjects by this definition, and the research was not subject to IRB review.

Furthermore, even if the project had been reviewed, it would have been exempt under 45 CFR 46.101, as "Research involving the use of educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures or observation of public behavior" with anonymous subjects. As such, at worst Prof. Boghossian should be admonished to seek IRB review for such research in the future.

Third was the accusation of "fabricating data" due to one of the fabricated research articles containing made-up statistics about canine sexual activity. Clearly, since the article was not intended to remain a part of the research literature, but to be unmasked as false, there was no intent to deceive the research community. As such, this was not fabrication or falsification of research data or results.

Taken together, the facts seem clear that Prof. Boghossian's project never warranted IRB review at all, or if it did, would have been exempt. In any case, the only potential consequence should be a discussion with him regarding the importance of undergoing IRB review for future such projects. I urge that Portland State rescind its determination that he violated university policy, and restore his professional and academic standing within the university to the status quo ante.

I would be happy of course to discuss this matter further if it would be of use.

Sincerely,

Shlomo Engelson Argamon
Professor of Computer Science
Director, Master of Data Science
Illinois Institute of Technology

Calls to Jettison "Statistical Significance"

I recently wrote an opinion piece for American Scientist's Macroscope blog, arguing (inter alia) that the notion of "statistical significance" harms the epistemology of science, and should therefore be jettisoned. After I'd submitted the piece, I saw other articles, letters, and posts saying the same thing, arguing from a number of technical and sociological bases. I will gather here, to the best of my ability, a list of these articles as a resource for the community. If you know of others that I've missed, please add them in the comments.

• Don’t Strengthen Statistical Significance—Abolish It, by Shlomo Engelson Argamon
• Abandon Statistical Significance, by Blakeley B. McShane, David Gal, Andrew Gelman, Christian Robert, Jennifer L. Tackett
• Justify Your Alpha: A Response to “Redefine Statistical Significance”, by Daniel Lakens, et al.
• Abandon Statistical Significance (blog post), by Andrew Gelman
• Remove, rather than redefine, statistical significance, by Valentin Amrhein and Sander Greenland
• Time to Abandon Statistical Significance, by Lars P. Syll

Perverse Incentives from the New "p < 0.005" Proposal

Here is a short addendum to my post of yesterday, commenting on why I think that the proposal recently mooted to require a p < 0.005 for "statistical significance" is misguided and likely problematic.

There is another problem with that (admittedly well-intentioned) proposal, which arises because wide-spread implementation of the proposal would create a perverse incentive which could easily degrade the quality of the scientific literature by much more than it improves it.

As the authors acknowledge, lowering the alpha needed to declare "significance" would require larger sample sizes, and so data collection would entail greater difficulty and expense. They estimate that samples would need to be 70% larger - that may well be for studies of a single variable, though research that involves interaction effects would likely need a greater increase in the amount of data. Regardless, the manpower and expense would increase notably. (The authors see this as a potential benefit, in that "considerable resources would be saved by not performing future studies based on false premises.")

However, this extra work and expense creates a perverse incentive for researchers. The more you pay (in time, effort, money, etc.), the more you want to recoup your costs by producing publishable research. The harder such results are to produce, the harder you will look for them, after you have sunk the costs of getting the data.

Now, this is not a bad thing, if there is no way to cheat unwittingly. (Let's ignore entirely the possibility of fraud.) But if it is still possible (as it is today) to unknowingly p-hack, traipse through the garden of forking paths, HARK, etc., raising the cost of gathering a dataset will inevitably lead to a rise in p-hacked, forking pathed, and HARKed results.

The very change that was meant to improve the quality and integrity of the research literature will rather act to degrade it.

It seems to me that this underscores the fundamental need to address how incentives interact with methodologies and standards. Otherwise we are, at best, spitting in the wind, and at worst, using gasoline to put out the fire.

New "p < 0.005" Standard Considered Harmful

Several days ago, an article preprint was posted on PsyArXiv (to appear in Nature Human Behavior), arguing that a significant step towards solving the reproducibility crisis in many fields will be to reduce the standard significance threshold to require p < 0.005 (rather than the current standard of 0.05) for a result to be termed "statistically significant". Results with higher p values could only be termed "suggestive" under this proposal.

I, along with a number of other scientists, was recently interviewed for a brief article in Nature | News which clearly, and concisely, sets out arguments for and against this proposal as a way to address the reproducibility crisis. I'd like to expand here on my very compressed comments as quoted in that article.

On the one hand, the proposal to reduce the "significance threshold" to 0.005 will indeed be helpful, in that the change will reduce the occurrence of false positives, and thus enhance the likelihood that effects termed "significant" will be reproducible, and so reduce the number of non-existent effects reported in the literature as real. 

However, the proposal does nothing whatsoever to change the fundamental problem, which is the entire framework of judging experimental results by a binary criterion of "significance". There are two key problems with the regnant view of "statistical significance": 

1. It does not account for effect size - a large effect with less-significant p may have much more "real-life significance" than a tiny effect with more-significant p, and
2. It is very easy to (even inadvertently) "p-hack" and find putatively significant effects due to the very large number of possible parameter settings for any experiment in a sufficiently rich and theoretically-underdetermined domain. 

Given that, this new recommendation may even do significant harm, in that improving research outcomes somewhat (by lowering false positive rates) without addressing the fundamental problem could lead to a false sense of security and entrench bad methodological attitudes. 

To expand on this, the problem I have with the new recommendation is that it maintains the exact same conceptual framework of statistical significance testing that has been a major driver of the replicability crisis. 

In statistical significance testing, a level of confidence (previously 0.05, per the new proposal 0.005) is defined, and then the question asked is: Is the probability that the null hypothesis produced the experimental results less than that confidence level? If it is, the null hypothesis is rejected, and the hypothesis is considered statistically significant.

This framework does not account at all for the question of effect size - an effect may be statistically significant, but practically speaking negligible and irrelevant. For example, often the null hypothesis is of the form "x = 0" for some model parameter x. Any such null hypothesis is almost certainly false, since the actual value of x is probably at least slightly different from 0, and so some statistically significant result can nearly always be extracted with enough data and ingenuity, even though the result may not be significant in any meaningful way in the real world.

More fundamentally, no matter what confidence level you choose, if there are enough different ways to design your experiment, it becomes highly likely that at least one of them will give a statistically significant result just by chance. And unless every single detail of the experiment is decided and registered ahead of time, it is virtually impossible to ensure, except by care and good judgement by the researchers, that no choices are made along the way that could (unwittingly) enhance the significance of the final experimental results, since there are so many different choices. This phenomenon was termed the "garden of forking paths" by Gelman and Loken, and illustrated nicely by XKCD. (This problem is inherent in any metric-based decision criterion for identifying effects, and is not specific to statistical significance testing.)

Finally, the goal and the strength of the "p < 0.005" proposal is that it will reduce the number of false positive results in the literature. However, by the same token, there will be fewer reports of "less statistically significant" results, which will cause overall problems for the science, due to the "file-drawer" effect. The problem is that the vast majority of negative results are never reported, which gives a biased picture of the state of knowledge. By lowering the p-threshold, the file-drawer bias will be even greater, since fewer results, with only lower p-values, are published.

So how should we proceed?

There is no simple answer to this question, as there are several interrelated problems: p-hacking, effect sizes, and literature bias, as I've mentioned above, but also the question of how the incentive structures of modern science may bias the creation and evaluation of new knowledge (the file-drawer problem is an example of this). All of these factors contribute to the "reproducibility crisis", and all need to be addressed. Any simple answer to one of these, such as changing the threshold p-value, will not solve the problem, and may in fact make things worse by making it seem solved when it is not.

Any meaningful solution must involve a combination of new methodological standards, institutional structures, and incentives. Reducing the p threshold is simple and could be one useful methodological standard, though it would perpetuate problematic concepts, as I've said. Another would be introducing some notions from Bayesian statistics, such as (i) using Bayes factors, which are more easily interpreted in most situations than a p-value, as a standard reported metric, or (ii) the use of explicit priors, to make assumptions clearer, or (iii) explicit loss functions, to account explicitly for the importance of effect size and the varying weights of different kinds of errors. 

It is essential that the scientific community have good incentives for publishing replications, especially those that fail, and for publishing negative results.  (And there cannot be penalties for early-stage researchers for doing so!) There need to be more and better institutional structures for preregistering studies, like clinicaltrials.gov, but set up with the appropriate community standards and consequences, so that they actually work.

Just as during the early and laborious development of effective scientific practices and institutions several hundred years ago, there is no royal road to solving the crisis. A great deal of hard work lies ahead of us, but past experience has shown that it will (eventually) pay off in more solid understanding of our world.

Initial Linguistic Analysis of the Shadowbrokers Texts

This is a repost of my forensic linguistic analysis of "The Shadowbrokers texts", as posted on the Taia Global blog on August 18, 2016.

This is an initial linguistic analysis of the texts from “The Shadowbrokers”, as posted on Pastebin, taken from the tumblr account. This is a qualitative analysis, looking at patterns of grammatical and orthographic errors, to examine the question of whether the author of the text is a native speaker/writer of US English. A quantitative analysis would help firm up and estimate the reliability of our conclusions. This analysis assumes that all the texts were written by a single individual.
There are a number of grammatical errors that are not usual in native speaker US English:

1. Omission of definite and indefinite articles (“a” and “the”)
2. Omission of  infinitive “to” (e.g., “I want get” instead of “I want to get”)
3. Omission of modal verbs “should” and “must” and auxiliary verb “will”
4. Elision of “it” in “it is ...”
5. Use of progressive form “is Xing” instead of present or past tense form “X” (e.g., “He is breaking” instead of “he breaks” or “he broke”)
6. Use of “are X” instead of “are Xing” or “X” (“they are go” instead of “they are going” or “they go”)
7. Tense confusion – use of base verb form instead of past tense

None of these errors is perfectly consistent – there are examples of correct usage for all of them. Indeed, some grammatical complex sentences are perfectly correct, such as “No other information will be disclosed by us publicly” and “Your wealth and control depends on electronic data.”  Most sentences, though, have multiple errors.
Evidence that the author is a native speaker trying to appear non-native:

1. Spelling. The spelling is entirely correct throughout, including some long and complex words such as “dictatorship”, “prostitutes”, and “consolation”.  If this had been achieved through the use of spell-checking software, we would have expected to see at least one “Cupertino” (choice of a correctly-spelled but contextually wrong word).
2. Inconsistent errors. Grammatical errors such as omitting the infinitive “to” or using “is breaking” to mean “breaks” result, in a non-native writer, from deeply held intuitions about how grammar works. The fact that errors 2, 3, 5, and 6 all occur inconsistently (they occur a majority of the time, but not by much) indicates that someone was inserting errors, rather than making them naturally.
3. Mutually inconsistent errors. Errors 5 and 6 are odd together – if the writer knows about the progressive (-ing) form, then why do they use it only sometimes, when using the auxiliary “is” or “are” with the verb?
4. Grammatical errors in idioms. There are a number of idioms that would be surprising for a low-skilled non-native speaker to use, and some of them are used with grammatical errors that a skilled English speaker would be unlikely to make. The most reasonable explanation, then, is that the errors were inserted by a native speaker after writing the idioms.  Examples include:
   1. “or [the] bid pump[s] [the] price up”
   2. “bidding war”
   3. “top friends”
   4. “go bye bye”
   5. “where [does that] leave Wealthy Elites”

The cumulative effect of these multiple lines of evidence leads to the conclusion that the author is most likely a native speaker of US English who is attempting to sound like a non-native speaker by inserting a variety of random grammatical errors.

In the (unlikely) event that the writer is, in fact, not a native English speaker, their native tongue is much more likely to be Slavic (e.g., Russian or Polish) than either Germanic or Romance.

Stop the Overheated Rhetoric; the Actual Executive Order is Bad Enough

I just read the text of the recent executive order that supposedly "Bans Muslims" and some of the relevant statutes. Unsurprisingly, much of the overheated rhetoric and clickbait headlines (even from reputable news outlets) are inaccurate. I believe it is essential, especially if you oppose the order, to be as precisely accurate as possible when writing about it. This is my analysis based on reading the original documents - I would welcome any corrections if I got anything wrong.

First, we must distinguish between the two main elements of the order. One comprises restrictions on entry to the US of certain visa holders, both temporary visas and immigrant visas ("green cards"). The other comprises restrictions on admission of refugees. These two sets of restrictions should not be confused, as they are not the same and have different effects.

1. The order does not "ban Muslims". It stops entry of citizens of any of a set of countries, most of whom are Muslim, and does not affect entry for most of the world's Muslims.
2. The order does not list a specific set of countries to be affected, but rather refers to the list of countries which are not eligible to be part of the visa waiver program, per 8 U.S. Code § 1187. As far as I can tell, this list came from the Obama administration. As far as I can tell, there is no actual evidence that Trump picked a specific list of countries whose citizens he wanted to ban, contra many memes out there. The supposed link to his business interests is pure speculation. Indeed, there are other Muslim-majority countries where he does not have business interests which are not on the list.
3. The order is being applied to green card holders, that is, legal permanent residents (LPRs). (Evidently, DHS initially did not apply it to LPRs, but the White House overruled them.) This is unconscionable, and perhaps illegal. Indeed, the US insists that green card holders maintain continuous residence in the US - they cannot be out of the US more than six months without potentially losing their status. It is an odious policy, and bad for the US in many ways, to keep out someone whose home, family, and livelihood are all legally in the US, perhaps for many years.
4. A great many talented and productive academics and graduate students here will now be unable to travel abroad, for fear of not being able to return - indeed, many have already been caught abroad by this order and cannot return to their jobs or their studies. This is real human suffering, and also highly negative impact on US science and technology. Harming US research capabilities will certainly not help increase American greatness.
5. It is also entirely unclear to me on what legal basis people holding valid visas can simply be barred from entry for no reason other than their country of citizenship, without revoking their visas (this may depend on the visa type). Issuance of such visas and the legality of entry to the US is governed by Congressional statute. It would seem therefore to fall within their purview. (Alas, Congress has allowed the Executive to appropriate rather too much of its power in recent years and I don't expect them to do much now.)
6. The immediate application of this order within hours to people who had no warning, who were en route to the US with valid visas, who live in the US (whether as legal residents, as students, etc.), does nothing to improve the security of the country. It does, however, create an enormous amount of human suffering.
7. The limit of 50,000 refugees per year is well within historical norms before 2016. It is therefore not outrageous, but certainly can be argued with, on the merits.
8. The prioritization of people persecuted for being members of minority religions is entirely proper and long overdue.
9. The suspension of the US Refugee Admissions Program (USRAP) is extraordinarily overbroad for its purpose, and will cause great human suffering, especially as applied to people already in transit. The clause enabling case-by-case refugee admission (jointly by the Secretaries of State and Homeland Security) is vague, and does not provide any sort of meaningful allowance for refugee admission.
10. The idea of reviewing policies and procedures for immigration and refugee screening is excellent. However, to suddenly and very broadly stop all entry to the country for huge numbers of people is absurd, and clearly not well thought out. The chaos in airports today bears this out.
11. Promulgating poorly-thought out orders with little-to-no interagency or legal review, and no pre-implementation planning, that then require multiple iterations of executive clarification and changes in implementation, is a recipe for instability and chaos. Such instability is, in my opinion, likely to reduce the security of the United States, possibly dramatically so.

Some thoughts on the 2016 election

Listen. Seek first to understand. Do not assume malice, or stupidity, or even ignorance, but start from an assumption that those you disagree with (even vehemently) are rational and intelligent, but may have good reasons for seeing things differently from you. Dismissing people out of hand will persuade no one. We have had too much of that on all sides - it creates rancor, and hardens people's political positions. That helps none of us.

We live in a large and incredibly diverse and now divided country. There are not just two Americas, there are many. Government should therefore be as local as possible, otherwise politics becomes a cage match to determine whose views will control everything. Federalism is a historically-validated way to allow for a diverse and yet civil and functioning polity.

The presidency has accumulated far too much power, and so presidential elections take on apocalyptic dimensions (on all sides). Don't think about what your choice would do with the power, think about what the other guy's choice would do with it. And then limit the power inherent in the office to reasonable proportions.

Supreme Court appointments have become a significantly distorting element in our presidential elections. One's choice of president should not hinge primarily on who he or she will appoint. Reform of the system is clearly needed. One reasonable suggestion is 18-year terms, ensuring that each president has a predictable number of appointments. There are other ideas, but something must be done. A president is more than a justice-appointer.

All of us, at this time, should take great care with what we say, whether out of pain and fear, or out of relief and joy. Let us all try to contribute to an atmosphere of reconciliation, of improved understanding of each other, and of rebuilding a political culture in which we need not agree but can work together.

Guccifer 2.0: Russian, not Romanian

Lorenzo Franceschi-Bicchierai recently interviewed (over Twitter) Guccifer 2.0,  who released documents hacked from the Democratic National Committee and claims to be the hacker of the DNC, concluding that he is probably not Romanian as he claims to be, and may very well be Russian. (Other evidence also points to the hack being perpetrated by the Russians.)

What, if anything, does Guccifer 2.0's language usage in the interview say about his likely national origin?

I performed a linguistic and stylistic analysis of Guccifer 2.0's English language responses in the interview. I examined all the clear errors in English grammar and style/usage, and considered whether each indicates that the writer is more likely a native Romanian or a native Russian speaker.

Summary of Results:
There are seven unusual (non-native) features in the English of the interview. Out of these seven, five clearly point to the author being a native Russian speaker, one weakly points in that direction, and the last says nothing. Hence we can conclude that the author is far more likely to be Russian than Romanian.

Detailed Analysis:

1. He refers to himself as a "women lover" in the plural.  In Romanian this would be "iubitor de femei" (instead of "femeie"), while in Russian it would be "любитель женщин".   Note that in Romanian, the phrase contains the preposition "de", corresponding to English "lover of women", while in Russian there is no preposition, as in the English phrase he used. Also, the odds of the Russian combination occurring as a phrase (based on Google searching the phrases with constraint site:ru and site:ro, respectively) is higher that of the the Romanian combination occurring as a phrase.  Thus the phrase is more likely a calque from Russian than from Romanian.
2. He writes "I've already told," instead of "I've already told you." This is a case of "pro-drop", where an argument to a verb (usually a pronoun) is dropped when understandable from context. In Russian, the equivalent sentence "Я уже сказал" (or "я уже ответил") is grammatically and contextually correct. In Romanian, the equivalent sentence, "Deja am spus" is technically correct, but it sounds odd and robotic to native speakers without the pronoun "ti" (you). This example is thus evidence that the writer is more likely Russian than Romanian.
3. He uses the word "deal" multiple times to mean "hack" or "operation" (e.g., "DNC isn't my first deal", "...all my deals"). In Russian, a deal is "сделка", business is "бизнес", an affair is "дело", an operation is "операция", an effort is "усилие", and an enterprise is "предприятие". In Romanian, these are, respectively, "afacere", "afaceri", "afacere", and "operație", "efort", and "afacere".  This may point towards Romanian, due to the similarity of several of the common translations for this concept, but more likely points towards Russian, due to the similarity between "дело" (delo) and "deal" (as well as "сделка" sdelka) - the English word "deal" is an false cognate of these words, that a Russian speaker may choose because of the similarity of sound.
4. He replies to a request to reply in Romanian as proof of his origin with "Man, I'm not a pupil at school." This is odd because of the use of the word "pupil" ("kid" would be more likely in colloquial English), the use of the preposition "at" instead of "in", and the start of the sentence with "Man, ...". 
1. In both Russian and Romanian, the phrase "pupil at school" ("ученик в школе" and "elev la școală", respectively) is slightly rarer than either "kid at school" or "boy at school". There is no clear difference here.
2. In Russian, both English prepositions "in" and "at" are generally translated by "в", while Romanian has different prepositions, "în" (in) and "la" (at), giving evidence for a Russian native language.
3. The odds of the phrase "I'm not" (Russian: "я не"; Romanian: "eu nu") being preceded by "Man, ..." (Russian: "Человек" or "Tоварищ,"; Romanian: "Omule,") in Russian is more than five times greater than in Romanian. This phrasing is evidence for a Russian native language over Romanian.
5. One of the most prominent features of Russian (and other Slavic-language) speakers writing in English is the general lack of articles (the words "a" and "the"), as those languages lack them. Romanian, on the other hand, as a Romance language, has both definite and indefinite articles. While many of the interview responses used English articles correctly, there were a number which did not (e.g., "I used [a] 0-day exploit" and "... the NGP VAN soft"). The inconsistency is not a complete proof of Russian (or Slavic) L1 authorship, but is strong evidence that the writer is a Russian speaker with relatively good English.

There are in total 7 oddities in the English text that may indicate the native language of the writer. Five out of the seven point clearly to Russian over Romanian as the native language, one ("deal") points weakly to Russian, and the last ("pupil") is inconclusive.

Overall, therefore, the linguistic evidence consistently points towards the writer being a native Russian speaker. It is also possible that the writer is a Romanian speaker who has studied Russian (often L2 features spill into a third language more than L1 features do); however the writer denied knowing any Russian, and so the most reasonable conclusion is that he is a Russian native speaker rather than a Romanian native speaker. This evidence, combined with the evidently problematic Romanian language use in the interview, indicates clearly that Guccifer 2.0 is a Russian pretending to be a Romanian. (Note that this says nothing about whether the hack was perpetrated by state-backed actors or an independent hacker or group.)

This analysis is quoted in the New York Times and Yediot Aharonot (in Hebrew).

UPDATE: 

Special prosecutor Mueller's July 13 indictment of 12 Russian GRU operatives on confirms that Guccifer 2.0 was a front persona run by Russian nationals for the GRU, per US intelligence assessments.

Open letter to Sharon Kugler, Yale University Chaplain

Dear Ms. Kugler,

As an alumnus of the university, I was greatly disturbed to read the recent (Aug 21) letter to the editor in the New York Times from Rev. Bruce Shipman, the Episcopal chaplain at Yale. In this letter, he lays the blame for the increasing incidence of anti-Semitic attacks and other incidents on the actions of the State of Israel. This is as puerile as it is repugnant.

No matter what one thinks about the actions of Israel’s government, only someone who was already an anti-Semite would respond by attacking Jews in the streets. A mob attacking Jews in a Paris synagogue is not a political protest. Someone shooting defenseless Jewish school children in Toulouse is not criticizing the Israeli government. It should be obvious to any thinking person that the inclination to such anti-Jewish violence preexisted any political excuse.

More fundamentally, however, Rev. Shipman’s argument is just like one blaming African Americans for racism because of high crime rates. What more need be said? It is a morally repellent claim which anyone, especially one considered a man of God, should reject with all force conceivable.

Reasonable people can certainly have differences of opinion as to the propriety and even morality of various governmental actions. But there should be no question that it is deeply immoral to use such opinions to justify hateful, racist violence.

The Yale Religious Ministries Agreement, which Rev. Shipman has signed, stipulates that members “will not undermine another faith community.” And yet he has done exactly that, by justifying anti-Semitism. This is an outrage, and he should be promptly expelled from the Yale Religious Ministries, as provided for in that Agreement.

Sincerely,

Shlomo Engelson Argamon (’94 PhD)