Assessing the Assessors

What did analysts get wrong about the Russian invasion?

The wreckage of Hostomel Airport - a key battleground on the first day of the Russo-Ukrainian War

People in my strange line of work spend a lot of time considering wars that probably will not happen, or if they do will come about in forms that had not been anticipated and with correspondingly unexpected outcomes. They face the familiar problems that come with trying to predict events that depend on confluences of circumstances. One can never weight the known factors accurately or be sure of their interaction with each other, let alone the unknowns, or account for the eccentric choices made by key players or the chance developments that can affect their calculations.

At any rate many of these expert debates on future wars are not really about prediction. They are more about influencing policy choices, from weapons procurement to diplomatic initiatives. The direst scenarios are drawn up to show what will happen if the wrong choices are made. I explored all this in a book published in 2017 which charted the history of the ‘Future Wars’ literature.

The incessant, ongoing discussions within and between government departments and think-tanks have about whether a Chinese invasion of Taiwan is likely soon are a good example. Some warn that this could happen any day now and others insist that China is not yet ready to pounce and may never be ready. It might have more subtle means of achieving its objectives. Over time these discussions can become routine with everybody’s position well known. But then with a jolt something changes, either in official rhetoric or force deployments or a combination of the two. What exactly is North Korea up to at the moment, ratcheting up its rhetoric and cutting off the road and rail links with the South? Anyone care to offer a prediction? At such times, when something is going on but what is less clear, the discussions become more urgent and focused. Established positions need to be checked against the new evidence and adjusted accordingly. It is a regular criticism of intelligence agencies and expert communities that they are bad at this. Hence the regular accusations of ‘groupthink.’

The most challenging set of debates in which I have ever been involved were those leading up the full-scale Russian invasion of Ukraine on 24 February 2022. These began in the second half of 2021 and reached a crescendo of speculation in the weeks leading up to the actual event.  The sheer scale of the Russian military buildup meant that it was impossible to ignore the possibility that Vladimir Putin was really planning to invade even though he and his minions denied the intent. The debate about whether this was real or a bluff continued almost up to the moment the first troops crossed the border. Because of the stakes, unsurprisingly, the debate became quite heated. Those who thought the invasion would go ahead did so not only because of the compelling evidence of Russian preparations but also because they thought that with the initiative and overwhelming strength Russia’s invasion would most likely succeed.

My own position was that it was impossible to dismiss the preparations as mere bluff yet it was unclear what the objectives would be and that with any such operation, whatever the apparent odds in favour, success should never be assumed. Military action is always full of pitfalls and rarely goes quite to plan. This did not reflect any detailed research on my part about the particulars of this conflict. It was hard to dismiss the Russia’s airpower advantage, so I did think that Ukraine would struggle to hold positions in the first stages of any war but then I also assumed Russia would struggle to occupy a hostile country. So for my part, this was more a belief about the nature of modern warfare than any expert analysis of the respective armies.

In the weeks after the invasion there was much soul-searching (or at least should have been) from both sides of the pre-war debate: those, like me, who thought Putin wouldn’t do such a foolish thing needed to ask why they had misread the signs that he was deadly serious; those who were sure there would be an invasion needed to ask why despite confident predictions the Russia were unable to defeat the Ukrainians with conventional forces in relatively short order.

This second debate, although not the first, has been re-opened with the publication of a paper by the Center for Strategic and International Studies (CSIS) in Washington DC, by Eliot Cohen, a senior fellow there, and Phillips O’Brien, of St Andrews University, with a lengthy foreword by Hew Strachan, also of St Andrews. The paper can be found here. The critique is long and detailed (there is a shorter version in the Atlantic). I should state that I am friends with all three having known them for many years. I am also friends with some of those targeted in this critique, including Mike Kofman, a senior fellow at the Carnegie Endowment, with whom I discussed these issues in a post on this site a year ago, to which I’ll return.

The issues raised in the CSIS paper are important and there is nothing wrong with some vigorous arguments, even amongst friends, on what can be learnt from this episode for the challenges we might face in the future, including over Taiwan. The question posed goes beyond the specifics of this case to the general question of whether there are some in-built biases which can lead analysts to over-estimate aggressors and fail to appreciate the possibilities of resistance. Can the analysis of what went wrong in 2022 help avoid similar mistakes in the future? There is a further issue surrounding the value of what are often described as ‘worst-case scenarios’ as they might serve as a spur to action but can also be a cause of fatalism and an excuse for passivity.

The Indictment

The CSIS charge is stark:

‘Their misjudgment was not a case of normal error or exaggeration. The expert community grossly overestimated Russian military capabilities, dismissed the chances of Ukraine resisting effectively, and presented the likely outcome of the war as quick and decisive.’

In the event the Russian air force was unable to suppress Ukrainian air defences and control the skies. Moving forward the Russian army suffered from problems in command and control and logistics that meant that they had difficulty sustaining their advances. By contrast Ukrainian conventional forces proved to be robust and adaptable.

The most straightforward part of the CSIS case is to show that there was a strong consensus view among those who studied the Russian military that it should have no trouble beating the Ukrainians in a conventional battle. This was backed up by studies of Russian equipment, doctrine, and recent exercises. It was also influenced to a degree by relative Russian successes in the Syrian war from 2015, by comparison with what had been poor performances in the Chechen wars and also in Georgia in 2008. It is also self-evidently the case that Ukrainian forces were underestimated.

The more difficult part of the critique is the suggestion that what eventually transpired could have been predicted. Cohen and O’Brien give short shrift to the idea that the Russians were just unlucky and that with better tactics they might have succeeded. The logic of their argument is that not only were the chances of Russian success exaggerated but that what did actually transpire should have been expected.

Here there is a tension with another part of their case - that there should more respect for the inherent unpredictability of battle:

‘Analysts portrayed war as a matter of equipment and doctrine, something that could be planned and executed, more as a complex engineering project than as a contest of will, military skill, and personalities.’

Account should have been taken of ‘friction, the fog of war, and the unpredictable nature of military interaction on the battlefield.’ This is also the issue addressed by Hew Strachan in his foreword, which says less about the specific of this case and more about the problems with conventional military analysis. He warns against comparative counts of equipment and manpower that fail to take account of factors which can’t be measured, such as morale an patriotism, on the conduct of operations, and the neglect of considerations such as terrain and weather.

As this was the basis of my pre-war scepticism I have some sympathy with this part of the critique. When I raised similar points prior to the war the response tended to be that the Russians had sufficient spare capacity to cope with unexpected problems, but also that as both sides would be subject to these factors they would balance each other out and not affect the outcome.

All of this assumed of course that the Russians knew what they were doing. It would have been quite a call to say that they did not, and that their plans were based on a poor assessment of the capabilities of their own troops as well as underestimation of the Ukrainians. But they did. So we have to keep in mind that that the most egregious errors were made not so much by the external analysts who were trying to make sense of what they were doing, but by the Russian general staff and political leadership who turned out to have lacked a true appreciation of the relative strengths of the opposing forces, and the way that chance factors might mess up plans. Thus before one gets to any analytical failure there is the prior question of Russian failure.

Why did the Russians fail?

Did the Russians fail because their armed forces simply weren’t good enough? Central to the early Russian failure was underestimation of Ukrainians. This was not purely a poor military assessment but also a poor political assessment. Putin had never hidden his views about the artificial nature of the Ukrainian state and the illegitimacy of its government. This was an important clue to the extent of the underestimation.

We now know at the heart of Russian planning was an FSB-operation to take out the Ukrainian leadership at both the state and local levels. There was mention of this in some pre-war analysis, as Russia would need a puppet government if it sought to occupy all of Ukraine. This was also relevant to the substantial side debate underway about whether it would make more sense for Russia to concentrate on taking the contested province of Donetsk and Luhansk, which would be militarily more straightforward, while leaving open the political future of Ukraine. So part of the challenge was anticipating the strategic ambition behind any Russian invasion and the extent to which this would shape the operational plans.

At this point let me turn to my interview with Mike Kofman. He describes the consensus view as being that ‘Ukraine would probably lose the conventional phase of the war, but that Russia would fail in a large-scale occupation of the country.’ He acknowledges overestimating ‘the Russian military’s ability to execute a large-scale effort like this’ and underestimating ‘the Ukrainian military, and civil society.’ He recognizes the importance of factors such as ‘will to fight’ and acknowledges the need to ‘get better at considering soft factors, and intangibles, but these remain difficult to measure and assess.’ So this much is agreed and should influence future analyses in comparable situations.

The main reason for the gap in expectations, Kofman suggests, was that instead of the Russian invasion looking like a conventional military campaign, the

‘invasion plan stemmed largely from the Kremlin’s assumptions, driven by a belief that Russian intelligence had set the conditions for a speedy coup de main. Extensive efforts were made at infiltration, subversion, and sabotage, but Russian intelligence failed to deliver. The Russian military structured the invasion assuming the groundwork had been laid, resistance would be weak and isolated, and that combat operations could be swiftly concluded.’

‘As one thoughtful Ukrainian colleague put it in a later discussion, they expected someone would “open the gates” for them. Hence little effort was made to isolate the theater, cut ground lines of communication, or engage in other actions that I think analysts anticipated as part of the operation. Russian forces did not conduct a combined arms operation, but essentially drove in, initially trying to conduct “thunder runs” along divergent axes of advance.’

From this perspective much responsibility for the failure lies with the FSB and whatever assurances they gave Putin about their successful subversion. Because they had failed to set the conditions for a takeover of Kyiv the Russia military soon faltered, leading to a wider system failure, affecting command and logistics. Connected with this is the battle for Hostomel airport on the first day of the war, when in a back and forth fight the Ukrainians were able to prevent the Russians holding the airport nearby Kyiv in a condition that would allow it to fly in transport aircraft carrying troops to move quickly onto the capital to topple the government.

On this matter Cohen and O’Brien are unconvinced. They present this argument as a ‘justification for analytic failure’, so that the case being made was that all would have gone well for the Russian military were it not for the ‘unpredictable and foolish intervention of President Putin and his advisers from the Federal Security Service (FSB).’ On Hostomel they argue that even if the Russians had been successful

‘is it right to assume that a Russian column charging into a dense, hostile urban area like Kyiv, filled with armed civilians and light infantry, would have done better than their counterparts in, say, the city of Mariupol in the south?’

Here I think they are overstating their case.

Their point is that the problems with Russian tactics, organisation, and logistics would have manifested themselves whatever had happened in the first few days, and perhaps that is the case. But that is also not the issue. Russia’s attempt to take the Ukrainian capital and if possible its president, and the role of the FSB in this, is not about civilian meddling with an amazing military plan, but an explanation for Russia’s strategic failure. It is not about finding an excuse for one part of the analytical failure but identifying another problem with the pre-war assessments. I remember at least one heated discussion in which I expressed my view that Russia would be unable to establish a puppet regime in Kyiv and was told that this was nonetheless part of the plan – we were both right. But I am not sure whether even those who recognised this as part of the strategy appreciated how much it would also condition the military campaign.

This points to a problem with the Cohen/O’Brien critique. There is a difference between observing that the Russians tried and failed and claiming that they were bound to fail whatever they did. Elsewhere in Ukraine, in the east and the south, Russian forces, at times helped by traitors in senior positions, did make rapid progress. Much of this territory has yet to be liberated. In Kyiv it made a significant difference to Ukrainian morale and international support that President Zelensky was able to demonstrate that he was alive and in charge. It might have looked different if he had followed American advice to move to a safer place to the west of the country.

Kofman also notes – which I remember hearing a lot about at the time – concern about the apparent lack of Ukrainian preparation. The government publicly dismissed US and UK warnings about Russian intentions, and to many this seemed dangerously complacent. We now know that rather belatedly but just in time Kyiv took steps, necessarily in secret, to prepare for the invasion so they were less caught by surprise than had been feared. If this mobilization had been delayed Ukraine would have been in more trouble. (By way of comparison think what might have happened if Israel had taken more seriously the evidence it had about Hamas’s plans).

Acknowledging the importance of chance and uncertainty in war therefore works both ways, potentially affecting the quality of Ukrainian as well as Russian decisions and their implementation. Similarly one might note that an analysis of the expectations surrounding the Ukrainian counter-offensive of 2023 might observe overestimation of Ukrainian forces and underestimation of Russian. (Again going back to the Middle East one might argue that the ability of Hamas and Hizbollah to resist the IDF militarily was consistently exaggerated though not their ability to resist politically).

Policy Implications

These are difficulties that anybody who takes a professional interest in these matters must recognise. Those with policy responsibility need to appreciate why their analysts may not always give them the certainty they crave or may mislead them if they try to do so. This can be frustrating for those who need to take decisions based on expectations about what might happen next, and might even hope to influence what happens next by taking the right policy decisions.

In this case while there was general agreement in the period leading up to the full-scale invasion that Russia was militarily superior there were differences of opinion about what this might mean in practice and certainly about the policy implications. One of the risks with an exercise of this sort is that individuals with disparate views are lumped together as if together they shared a coherent set of beliefs about the situation. This was not a homogenous group. The US administration warned in the starkest terms about Russian intentions and capabilities from late in 2021. Part of the objective was to impress upon the Ukrainian government the danger they faced and to urge them to step up their defences. Other analysts expressing concerns about Ukrainian vulnerabilities were working closely with Ukrainian commanders in country to find ways to address them. By contrast others pushed a different message (highlighted by Cohens and O’Brien) that it was pointless for the Ukrainians to attempt resistance and that it would only make things worse if they tried.

What would have been the contrary policy implications connected with a ‘best case’ view that Russian forces were bound to fail? Perhaps to stay well clear and let the Russians stumble into their quagmire, or else to give the Ukrainians even more support so the Russian failure could be more substantial. But a decision to help the Ukrainians more would have led to the same arguments with which we have become familiar about NATO getting blamed for Ukrainian success and the risks of escalation.

Cohen and O’Brien propose remedies to the deficiencies they identify. These are sensible – you need many diverse viewpoints, including generalists who know something about military history, as well as specialists who know the countries. Problems can develop if the expertise is too weighted to one perspective or of there is insufficient challenge. But it is also important to ask the right questions. Looking back there was no shortage of debate, and from different perspectives. Few people appeared not to have an opinion on Russian intentions. The issue of the day was not, however, would a Russian invasion fail but would Russia invade in the first place.

And yes there should be some accountability. Cohen and O’Brien want outsiders

‘to keep book—not with the purpose of banishing or blacklisting analysts but confronting them with their errors and putting them in a position to reflect on why the errors were made.’

In my experience there are plenty of people prepared to call out the errors of others, especially on social media, although they are not always disinterested seekers of truth themselves. It is more important for anyone who regularly opines on these issues to take responsibility for their assessments. I now make it my practice to mark each new year with an assessment of what I got right and wrong in the previous twelve months (here for 2022 and 2023). It is always a salutary experience. In practice, without access to the front lines or to the best classified information on the state of the war and future plans it is always going to be difficult forecasting future developments. That is why I concluded some time ago it is probably best not to do so or, at least, not without qualifications and caveats. There are ways of talking about future possibilities without getting over-invested in particular scenarios.

But I do think it is important for those who can contribute to these debates to do so. This should be done with care and humility, aware that some contributions may look awful with hindsight. But as it is a very rare analyst who gets everything right, we are only going to get a diverse and lively debate if we accept that many of those contributing are going to get things wrong.