Tag Archive for: Tender evaluation

In part 1 here I discussed the reports that Camelot, the current operator of the UK National Lottery, is going to challenge the government’s decision to award the contract for management of the Lottery to a different firm, Allwyn, headed by a Czech tycoon. That decision follows a lengthy and no doubt exhaustive “procurement” process.

There are suggestions that Allwyn have offered to make more money for charitable causes than Camelot included in their proposal. According to reports, that amount is not contractually  guaranteed, but may have played a major role in the selection decision.  Which leads us into the question of confidence – how do we know that supplier really will deliver what they promised?

There was a great comment on LinkedIn related to the part 1 article. The writer told of a major NHS procurement where a US supplier came in with a knockout bid, which led to other potential suppliers simply pulling out. Then, literally on the day the new service was due to go live, “At the eleventh hour the supplier had withdrawn, admitting that they couldn’t deliver the brief and make the savings claimed”.

There is a huge difference between what suppliers (some suppliers at least) will claim they can do and what they actually can deliver. There are no magic answers to this, but in my book “Bad Buying” I suggest thinking about “analyse, reference, test”.

Analyse means looking into the firm, the product or service that you’re going to buy, doing your research on the supplier and on whatever you are buying. The amount and depth of research needs to be proportionate to how much you’re spending and how critical what you’re buying is.

Reference means asking other customers of your potential supplier or users of the product or service you are buying about their experience. It’s an obvious step, yet it is amazing how many organisations don’t bother with this step. I was asked for input on a legal case in 2018 where an incumbent supplier challenged the decision by a large government body to award a contract to several other firms, meaning that the incumbent was going to lose all its business. This was a really sensitive service; if it went wrong, you might well see reports on newspaper front pages.

Yet when the incumbent firm asked questions about how the procurement decision was made, it became clear that the government organisation had done virtually nothing to check out what other suppliers were claiming in their bids. They had not researched the track record of the firms; they had not taken up references from other customers; they did not even seem to have checked whether the directors of bidding firms had criminal records! The buyer was simply believing the bidders and hoping for the best. The competition was eventually re-started as I assume the lawyers told the contracting authority they were going to lose in court.

Test means using techniques such as pilot programmes or small-scale rollouts that enable you to get a sense of the supplier and their capability, without immediately betting the farm on a particular approach. In a large organisation, you could run a geographical experiment with a new supplier or product. Give it a try in an area, region, an office or a factory, rather than moving immediately to handing over your entire business. Or you might initially use a supplier on a relatively unimportant piece of work.

In the case of the lottery, I assume that Allwyn’s references have been thoroughly checked out. Perhaps most critical – if this comes to court – will be how the projections of the money to be made for charity have been developed and verified. I’m sure the buyer would be expected to analyse Allwyn’s assumptions and proposals very carefully to assess the level of confidence in their figures. If they did not, that could spell trouble.

The final point to make here is that one report quoted Camelot as saying the evaluation had not been carried out as described in the tender. Now if that is the case, the lottery folk are in real trouble.

In terms of public sector tender evaluation, not doing what you told the bidders you would do is in most cases enough for a challenge to succeed.

You simply can’t introduce new factors once bids have been received evaluation; or even use factors that aren’t explicit. Don’t make assumptions. You can’t mark down a bidder for not providing a detailed quality plan if your question simply said, “tell me how you will deliver this work”. If the quality plan matters, tell them to provide it.

Enough of my ranting about evaluation processes (a favourite topic of mine, and we haven’t even got onto evaluating and scoring “price”). We will await the next stage of the Camelot story with interest.

Over the weekend, we saw reports that Camelot, the current operator of the UK National Lottery, is going to challenge the government’s decision to award the contract for management of the Lottery to a different firm, Allwyn, headed by a Czech tycoon. That decision follows a lengthy and no doubt exhaustive “procurement” process. This is from The Times (behind the paywall unfortunately).

The Czech bid, led by Sir Keith Mills, the man behind the London 2012 Olympics, and the former J Sainsbury boss Justin King, was deemed to have had an inferior business plan but managed to pip Camelot at the post by promising to deliver a much higher sum for good causes. There are suggestions that Allwyn’s bid was based on a forecast that it would raise £38 billion over the ten-year licence, which starts in 2024. This is believed to be a much higher figure than the forecast included by Camelot … Bidders were asked to supply a detailed forecast of how much they expected to raise, but with no obligation to achieve it or any form of penalty for failing to do so.

So this may come down to an issue that sits behind one of the common causes of “Bad Buying”.  In my book of that title it has its own chapter – “Believing the Supplier”.

That can relate to suppliers actually lying or deliberately misleading the buyer. It’s the tech firm that says they can develop and install the new software for you in six months, when they know its going to be more like 18.  Or the consulting firm that tells you they have lots of experience running M&A studies in Spain, when in fact they have one junior analyst in the London office who has a girlfriend in Madrid.

But more often it is suppliers whose intentions are good, but make promises and offers that they can’t really deliver on. They really do believe that software will be ready in six months; but they don’t actually have the experience or expertise to make it happen.

This leads to a particular issue in public sector procurement. Because that relies on formal tendering processes (for larger contracts anyway), we see a real difficulty for buyers in assessing two different aspects of the proposals received. They have to evaluate the apparent value of the solution proposed, which is what the legal procurement framework focuses on. But they should also assess the credibility of the proposal – the confidence the buyer can have in its actual delivery.

You might remember the “scandal” back in 2008 when the UK’s Qualifications and Curriculum Authority (QCA), which managed the school ‘national curriculum’ and associated testing process, terminated a contract it had put in place with ETS Europe to deliver tests for schoolchildren.  ETS failed to meet agreed timescales and the whole thing was a bit of a shambles.

The case illustrated a central challenge in many buying situations – how the buyer can assess whether proposals can actually be delivered by a potential supplier, even if they sound credible. It is relatively easy to write a convincing proposal to carry out services-type work or even to deliver certain physical items. I might tell you in beautifully written prose that my firm can supply you with the finest cocoa beans, or handle your outsourced pension administration absolutely brilliantly. Or even build a nuclear submarine … But how do you know I can actually do it? Here is an extract from Bad Buying that explains what went wrong with ETS, following an independent review into the case. 

“The Sutherland Review found that in many ways the procurement (buying) process in this case wasn’t run badly– the authors called it ‘sound’. ETS won with the lowest price, but also scored better than the alternative bidder on non-cost factors. The ‘Gateway reviews’ undertaken by the Office of Government Commerce were in general positive, too. However, the contract and the supplier clearly failed to deliver what was required. Why was that?

Issues were identified by the report around governance, the contract- management approach, some legal issues in the contract and specifications. But the report suggests that the weakness in the selection process came from two key factors. First, the QCA and the consultants running the process did not fully check out the history of previous contracts delivered by ETS. That might have picked up warning signals, as there had been issues with contracts in the United States. Basic financial health checks were done, but not an extensive reputational and performance due diligence.

Second, the buying process did not check that the assumptions about capacity made by ETS in their bid were realistic and accurate. The firm should have been challenged more strongly on its staffing plans. There were also concerns about the ‘end-to-end’ solution proposed and whether the firm really understood how different elements needed to fit together. Those issues appear to have been at the heart of subsequent problems.”

So it is this “confidence in the supplier’s ability to deliver” that has to be assessed somehow, and whether the supplier’s assumptions and plans are “realistic and accurate”. It is not just their conformance to the specification, the elegance of their proposed solution, or indeed the apparent financial benefits they might be offering.  

Going back to where we started, it is this issue that may come to the fore in the UK National Lottery case, assuming the decision is challenged. More on that in part 2.

(I asked CCS if they wanted to comment on this article and they said no).

The Crown Commercial Service (CCS) is the central buying organisation for the UK government – particularly used by central departments, although any other public body (councils, hospitals, universities etc.) can use their contracts and frameworks too. It does some good work and employs a lot of hard working, smart procurement people. But sometimes it gets it badly wrong, as it has with the new management consulting procurement process.

Bids from potential suppliers are now in for the latest iteration of their Management Consultancy Framework, MCF3 as it is known. It is split into 10 Lots, ranging through general “business”, functional areas including procurement, and high-level topics such as “strategy”. Suppliers can bid for all or any of the Lots.

I have looked at the way the Lots and evaluation process are structured, and the way it is designed looks at first sight very strange. However, if you believe that it is aimed at meeting four key objectives, then it is quite sensible. Those objectives do not, unfortunately, include “delivering value for the taxpayer”.  

Instead, they appear to be:

  1. Make sure the big firms (McKinsey,  Deloitte, BCG, PWC etc) win a place on the more “strategic” Lots 2, 3 and 4 for strategy, finance and transformation work.  Why is it essential that these firms are successful?  Simply because Ministers and senior civil servants want to use those firms, and CCS itself relies on the commission it gets from sales through its frameworks to fund itself. If they weren’t available via CCS, budget holders would find another way to engage those firms and CCS would lose revenue.  
  2. Make sure those firms get onto the framework without having to offer particularly competitive prices, so they will be happy to put senior people onto government work without worrying about the rates.  
  3. Ensure that there are a large number of “SMEs” (smaller firms) who win a place on the MCF. Ministers can then supposedly support the small business agenda and announce that “over 50% of the firms selected are small firms”.
  4. But also make sure there is no need for any government department to actually use any of these small, lower cost firms.

So if these are indeed the objectives, how has CCS given itself the best chance of achieving this?

A Dodgy Price Evaluation

The way price is evaluated is a major factor here. So Lot 1 is general “business”, and up to 75 suppliers will be appointed to this Lot. Here, when the bidders “price” is evaluated, it is weighted at 90% of the total marks available. But the other 10% is just a tick box to say you will deliver the services (which is odd in itself – why would I be bidding otherwise?)

Price is calculated as the median of the prices offered for the 6 grades, from junior consultant up to Partner level. So basically, this is purely a price selection. The cheapest firms, which will be small firms that few of us will have ever heard of, will win a place. And because no-one has heard of them, and (in some cases at least) they are not very good, which is why they are cheap, they won’t be used much. But CCS and Ministers will have lots of SMEs on the list to boast about.

So then how does CCS make sure that the big firms succeed? For Lots 2, 3 and 4, price is only weighted at 10% of the total marks.  The rest come from essay-type questions in which the firms have to show extensive capability. There is plenty of scope for some flexibility in the marking too, and given the low weighting, price barely matters.   I would bet my mortgage that the “usual suspects” will all win places here.  

But just to make sure that those firms don’t have to worry about not making enough money, the price on which marking is based is not calculated as the average (the mean) of the 6 grades, which would seem to be a logical approach, or a weighted average rate based on likely frequency of use of each grade. Instead, it is the average of just two grades, the two “middle” ones (senior consultant and principal consultant / associate director). Actually, that would seem to be the same as the “median” price which is how Lot 1 is defined – it is not clear why different terminology is used.

So that means you don’t have to worry much about the price you put in for Partner. There is one more constraint in that for each grade, the price must be between 10 and 50% lower than the grade above.  But that isn’t too much of a hardship – for instance, you could put in this bid:

FIRM A

Partner                               £6000 a day

Director                              £3000

Principal consultant        £1500

Senior consultant             £1300

Consultant                          £1150

Analyst                                £1000

Your score would be based on the average of £1300 and £1500, so that is £1400, which is probably not too out of line with many bids. But once you win a contract, you can legitimately put your Partners in at £6K a day!

This is an “Illegal” Evaluation Methodology

There is also a technical/legal issue here, in that your evaluation score could be the same as another bid that puts in much lower rates for the top two grades (or indeed the lowest two), as long as you offer the same rates for those two in the middle. That seems to break fundamental rules of public procurement, that you have to make “value” your selection factor and you have to show you have a “fair” process.  So Firm B (below) scores fewer points in the evaluation than firm A, even though their pricing is much better value overall!

FIRM B

Partner                               £2000 a day

Director                              £1750

Principal consultant        £1500

Senior consultant             £1320

Consultant                          £900

Analyst                                £600

I can think of no reason why the average of the 6 grades has not been used – other than to help the big firms charge a fortune for their Partners. Unless I’ve missed something here, it feels like either a real error or there is something odd going on. I’m not a conspiracy theorist, but you do sometimes wonder if there is some sort of plan for certain firms to suck as much money as possible out of the public purse at the moment?   

This is the Argos Catalogue, not a “Framework”

Finally, there is another somewhat technical issue, in that users of the framework who want to choose a supplier for an individual project should (to be legally compliant) in most cases invite all the suppliers listed in the Lot to bid. But if you have to ask 75 firms (Lot 1) or even 30 firms (Lots 2 to 6 ) to put in proposals, that is quite a workload to manage and evaluate.

So I suspect CCS assumes that many users will just choose their favourites from the list, even if this technically breaks the regulations. We’re going back to the old days when I worked in government in the 1990s and that was how frameworks were generally used. Budget holders just picked their favourites from a preferred supplier list. The approach didn’t deliver value for money for the taxpayer then, and it doesn’t now.

But again, having such extensive lists of suppliers ensures that there is plenty of choice on the framework for users, so CCS maximises its own revenues. I’m afraid that looks like a major driver here, along with keeping Ministers, budget holders and the big firms happy.

What Does the Lord Think?

I do also wonder what Lord Agnew, the Cabinet Office Minister, thinks of this, or if he is even aware of what is going on. It was Agnew who wrote to senior civil  servants last September telling them to “rein in spending on consultants” and that Whitehall was being “infantilised” by their over-use. 

But when you see headlines in a year or two about “firms charging £6K a day for consultants”, you know why. Basically, the government, through Crown Commercial Service, has designed its procurement process to allow that. This is all very disappointing, given the undoubted talent of the people in CCS involved in this exercise.

PS  Buying consulting services based on a “day rate” model is almost always the wrong way to do it, anyway. More on that another day.

PPS There is no mention of “social value” in the tender either.