Tag Archive for: Tender evaluation

,

Too Soon for 30% Weighting for Social Value in Tenders

I spoke recently at the UK Universities Procurement conference and as usual, had some interesting conversations around the margins of my session. In one such discussion, a sustainability person from a major university told me that his organisation was looking to increase the percentage of marks awarded to “social value” in tenders from 20% to 30%.  I must admit this surprised me, and I am certainly not in favour of this at the moment. It feels like we are heading for another new category of Bad Buying stories – where firms win tenders based mainly on their social value proposals rather than on their capability and the real “value” of their offering.

I have been consistently in favour of including social value in public procurement. But we haven’t been doing it for long, and I have not seen much analysis of exactly how successful it has been to date. So it seems too soon to be putting quite so much emphasis on that at the expense of cost, wider quality or service issues, supplier innovation and so on.  I would personally like to see 10-15% of the marks allocated to social value until we have more evidence.

One key concern is that organisations in my experience sometimes don’t really understand their own evaluation processes. My question to anyone thinking of moving to 30% is this. Given the evaluation methodology you are using, how much more are you prepared to pay for a proposal that scores 100% on social value creation as against one that scores 50%? Because that is what your evaluation scheme actually determines.

Some might say “ah, but social value has a real financial benefit too”.  In general, that is simply not true – certainly for the contracting authority itself. Read my article from a year ago here if you want more to support my claim). A quick extract – “In almost all cases, this is not real money. “Wooden dollars” as someone described it to me recently. It does not show up on the buyer’s P&L or balance sheet. You can’t spend these “financial” benefits on more road maintenance, a new operating theatre, or re-opening a drop-in centre for vulnerable people. No cash appears in the CFO’s hands.

The other big problem is that where there are benefits from social value, they often don’t go to the actual buyer. So if a university is accepting something like “employing more apprentices” as a positive social value factor, then how exactly does that benefit the university itself? Maybe it is good for society more generally, although big firms always employ apprentices so whether this is real incremental benefit from this contract is often questionable. We are also building in a barrier for smaller suppliers when we do this.

If we go down the 30% route, I can see some scandals emerging where contracting authorities end up paying way over the odds for goods or services, and their defence is “but the social value was great – look, the supplier painted a scout hut”. Yes, but was that worth the extra million you paid to a supplier who turned out to be not very good at the core work?  Look at the Scottish ferries fiasco if you want an example of what can happen when a basically incompetent supplier wins a contract for non-value for money reasons.

I don’t want to become an “anti-social value” campaigner, but I really don’t like the idea of 30% of evaluation marks going on social value until we understand a lot more about best practice and how we can get the most out of this initiative for the taxpayer. And we’re not there yet.

However, there is one more innovative option. You could specify a fixed price and then evaluate on service, social value and other factors. I have heard of this being done and it has some merits. So you might say “we are prepared to pay £500K for this service – now tell me how you will do it and what social value you will provide”. In that case, I’m open to a 30% weighting.

/0 Comments/by
, , , ,

Good Tender Documents – More Important Than You Might Think

You may have read about the recent UK hospital trust tender that hit the media because of its questions about diversity and transgender issues. It turned out that the questions should not have been included in the document; it was human error rather than anything else.

I recently got involved with another National Health Service tender – we’re talking about a “collaborative buying” framework here, potentially worth hundreds of millions.  A consulting firm I’ve worked with over the years asked me to look at the tender documents, because they could not work out how on earth the buyer could possibly differentiate between the various bidders. Basically, there were no evaluation questions that actually asked the bidders to explain their core technical capability!

I read it and agreed that is was a very odd document.  No selection outcome could possibly have stood up to legal challenge, for a start. Luckily, I knew a senior procurement person in the buying organisation, so I called and explained the issue. A few days later, the tender was pulled. Pure human error again.

I was reminded of these cases during an Oxford POGO session last week. (POGO is a very worthwhile knowledge sharing club – more details here). The topic was capability in public procurement, and there were a number of interesting speakers. But it was Steve Schooner, Professor of Government Procurement Law at the George Washington University Law School in Washington, USA, who brought up the issue of writing tender documents.

Too often that was seen as a pretty unimportant task, but he said (quite correctly) that is a key skill if you want to get the best potential suppliers, the best proposals and ultimately the best outcomes from your procurement and suppliers.

He also said that “no-one should be allowed to write a public sector tender document until they have sat supplier side and responded to a tender”!

I think that is a great idea and maybe should be a core training activity for developing public procurement professionals. Over the last decade or more, I’ve occasionally supported clients who were responding to (usually public sector) tenders. It has given me a lot of insight into what good procurement practice looks like – and more depressingly, what bad practice looks like. I’ve also worked buy-side of course and tried to help buyers to get it right! It is not always easy, but it is always important.

As well as the contribution of this stage in the process in terms of final outcomes, there is another factor to consider. The tender documents you issue are probably the most direct and often the most widely-read manifestation of your procurement function’s competence.  

You can claim to be a world-class team, you can win lots of awards, but if potential suppliers read your tender and think “what a load of old rubbish this is”, then more than anything that will be what informs their view of you. The same often applies with internal stakeholders. If there are non-procurement colleagues involved in a procurement process, and they see that the procurement professional doesn’t know how to produce good material, or (even worse) the stakeholder starts to get calls from frustrated potential suppliers, then this is very bad news for your internal reputation.

Going back to the beginning, I spoke to a senior person involved in the “controversial” case of the diversity questions. We’ve learnt two things, he said. Firstly, we need more and better training for all our staff who are involved in producing tender documents. And secondly, “we need better quality assurance before material goes out of the door”.

Often top procurement executives feel they are too busy to read tender documents, or that it is  a low-value task for someone of their seniority, skills and experience. Below their pay grade, as it were. But if that is your view, just remember – a lousy tender document has the potential to trash your team’s reputation more widely and faster than just about anything else.   

/0 Comments/by
, , ,

Will My Supplier Deliver? Lotteries, Camelot and Challenges

In part 1 here I discussed the reports that Camelot, the current operator of the UK National Lottery, is going to challenge the government’s decision to award the contract for management of the Lottery to a different firm, Allwyn, headed by a Czech tycoon. That decision follows a lengthy and no doubt exhaustive “procurement” process.

There are suggestions that Allwyn have offered to make more money for charitable causes than Camelot included in their proposal. According to reports, that amount is not contractually  guaranteed, but may have played a major role in the selection decision.  Which leads us into the question of confidence – how do we know that supplier really will deliver what they promised?

There was a great comment on LinkedIn related to the part 1 article. The writer told of a major NHS procurement where a US supplier came in with a knockout bid, which led to other potential suppliers simply pulling out. Then, literally on the day the new service was due to go live, “At the eleventh hour the supplier had withdrawn, admitting that they couldn’t deliver the brief and make the savings claimed”.

There is a huge difference between what suppliers (some suppliers at least) will claim they can do and what they actually can deliver. There are no magic answers to this, but in my book “Bad Buying” I suggest thinking about “analyse, reference, test”.

Analyse means looking into the firm, the product or service that you’re going to buy, doing your research on the supplier and on whatever you are buying. The amount and depth of research needs to be proportionate to how much you’re spending and how critical what you’re buying is.

Reference means asking other customers of your potential supplier or users of the product or service you are buying about their experience. It’s an obvious step, yet it is amazing how many organisations don’t bother with this step. I was asked for input on a legal case in 2018 where an incumbent supplier challenged the decision by a large government body to award a contract to several other firms, meaning that the incumbent was going to lose all its business. This was a really sensitive service; if it went wrong, you might well see reports on newspaper front pages.

Yet when the incumbent firm asked questions about how the procurement decision was made, it became clear that the government organisation had done virtually nothing to check out what other suppliers were claiming in their bids. They had not researched the track record of the firms; they had not taken up references from other customers; they did not even seem to have checked whether the directors of bidding firms had criminal records! The buyer was simply believing the bidders and hoping for the best. The competition was eventually re-started as I assume the lawyers told the contracting authority they were going to lose in court.

Test means using techniques such as pilot programmes or small-scale rollouts that enable you to get a sense of the supplier and their capability, without immediately betting the farm on a particular approach. In a large organisation, you could run a geographical experiment with a new supplier or product. Give it a try in an area, region, an office or a factory, rather than moving immediately to handing over your entire business. Or you might initially use a supplier on a relatively unimportant piece of work.

In the case of the lottery, I assume that Allwyn’s references have been thoroughly checked out. Perhaps most critical – if this comes to court – will be how the projections of the money to be made for charity have been developed and verified. I’m sure the buyer would be expected to analyse Allwyn’s assumptions and proposals very carefully to assess the level of confidence in their figures. If they did not, that could spell trouble.

The final point to make here is that one report quoted Camelot as saying the evaluation had not been carried out as described in the tender. Now if that is the case, the lottery folk are in real trouble.

In terms of public sector tender evaluation, not doing what you told the bidders you would do is in most cases enough for a challenge to succeed.

You simply can’t introduce new factors once bids have been received evaluation; or even use factors that aren’t explicit. Don’t make assumptions. You can’t mark down a bidder for not providing a detailed quality plan if your question simply said, “tell me how you will deliver this work”. If the quality plan matters, tell them to provide it.

Enough of my ranting about evaluation processes (a favourite topic of mine, and we haven’t even got onto evaluating and scoring “price”). We will await the next stage of the Camelot story with interest.

/0 Comments/by
,

Camelot Challenges UK Lottery Decision – Highlights Key Issue of “Confidence”

Over the weekend, we saw reports that Camelot, the current operator of the UK National Lottery, is going to challenge the government’s decision to award the contract for management of the Lottery to a different firm, Allwyn, headed by a Czech tycoon. That decision follows a lengthy and no doubt exhaustive “procurement” process. This is from The Times (behind the paywall unfortunately).

The Czech bid, led by Sir Keith Mills, the man behind the London 2012 Olympics, and the former J Sainsbury boss Justin King, was deemed to have had an inferior business plan but managed to pip Camelot at the post by promising to deliver a much higher sum for good causes. There are suggestions that Allwyn’s bid was based on a forecast that it would raise £38 billion over the ten-year licence, which starts in 2024. This is believed to be a much higher figure than the forecast included by Camelot … Bidders were asked to supply a detailed forecast of how much they expected to raise, but with no obligation to achieve it or any form of penalty for failing to do so.

So this may come down to an issue that sits behind one of the common causes of “Bad Buying”.  In my book of that title it has its own chapter – “Believing the Supplier”.

That can relate to suppliers actually lying or deliberately misleading the buyer. It’s the tech firm that says they can develop and install the new software for you in six months, when they know its going to be more like 18.  Or the consulting firm that tells you they have lots of experience running M&A studies in Spain, when in fact they have one junior analyst in the London office who has a girlfriend in Madrid.

But more often it is suppliers whose intentions are good, but make promises and offers that they can’t really deliver on. They really do believe that software will be ready in six months; but they don’t actually have the experience or expertise to make it happen.

This leads to a particular issue in public sector procurement. Because that relies on formal tendering processes (for larger contracts anyway), we see a real difficulty for buyers in assessing two different aspects of the proposals received. They have to evaluate the apparent value of the solution proposed, which is what the legal procurement framework focuses on. But they should also assess the credibility of the proposal – the confidence the buyer can have in its actual delivery.

You might remember the “scandal” back in 2008 when the UK’s Qualifications and Curriculum Authority (QCA), which managed the school ‘national curriculum’ and associated testing process, terminated a contract it had put in place with ETS Europe to deliver tests for schoolchildren.  ETS failed to meet agreed timescales and the whole thing was a bit of a shambles.

The case illustrated a central challenge in many buying situations – how the buyer can assess whether proposals can actually be delivered by a potential supplier, even if they sound credible. It is relatively easy to write a convincing proposal to carry out services-type work or even to deliver certain physical items. I might tell you in beautifully written prose that my firm can supply you with the finest cocoa beans, or handle your outsourced pension administration absolutely brilliantly. Or even build a nuclear submarine … But how do you know I can actually do it? Here is an extract from Bad Buying that explains what went wrong with ETS, following an independent review into the case. 

“The Sutherland Review found that in many ways the procurement (buying) process in this case wasn’t run badly– the authors called it ‘sound’. ETS won with the lowest price, but also scored better than the alternative bidder on non-cost factors. The ‘Gateway reviews’ undertaken by the Office of Government Commerce were in general positive, too. However, the contract and the supplier clearly failed to deliver what was required. Why was that?

Issues were identified by the report around governance, the contract- management approach, some legal issues in the contract and specifications. But the report suggests that the weakness in the selection process came from two key factors. First, the QCA and the consultants running the process did not fully check out the history of previous contracts delivered by ETS. That might have picked up warning signals, as there had been issues with contracts in the United States. Basic financial health checks were done, but not an extensive reputational and performance due diligence.

Second, the buying process did not check that the assumptions about capacity made by ETS in their bid were realistic and accurate. The firm should have been challenged more strongly on its staffing plans. There were also concerns about the ‘end-to-end’ solution proposed and whether the firm really understood how different elements needed to fit together. Those issues appear to have been at the heart of subsequent problems.”

So it is this “confidence in the supplier’s ability to deliver” that has to be assessed somehow, and whether the supplier’s assumptions and plans are “realistic and accurate”. It is not just their conformance to the specification, the elegance of their proposed solution, or indeed the apparent financial benefits they might be offering.  

Going back to where we started, it is this issue that may come to the fore in the UK National Lottery case, assuming the decision is challenged. More on that in part 2.

/0 Comments/by
, ,

Money No Object! UK Government Invites Proposals from Consulting Firms

(I asked CCS if they wanted to comment on this article and they said no).

The Crown Commercial Service (CCS) is the central buying organisation for the UK government – particularly used by central departments, although any other public body (councils, hospitals, universities etc.) can use their contracts and frameworks too. It does some good work and employs a lot of hard working, smart procurement people. But sometimes it gets it badly wrong, as it has with the new management consulting procurement process.

Bids from potential suppliers are now in for the latest iteration of their Management Consultancy Framework, MCF3 as it is known. It is split into 10 Lots, ranging through general “business”, functional areas including procurement, and high-level topics such as “strategy”. Suppliers can bid for all or any of the Lots.

I have looked at the way the Lots and evaluation process are structured, and the way it is designed looks at first sight very strange. However, if you believe that it is aimed at meeting four key objectives, then it is quite sensible. Those objectives do not, unfortunately, include “delivering value for the taxpayer”.  

Instead, they appear to be:

  1. Make sure the big firms (McKinsey,  Deloitte, BCG, PWC etc) win a place on the more “strategic” Lots 2, 3 and 4 for strategy, finance and transformation work.  Why is it essential that these firms are successful?  Simply because Ministers and senior civil servants want to use those firms, and CCS itself relies on the commission it gets from sales through its frameworks to fund itself. If they weren’t available via CCS, budget holders would find another way to engage those firms and CCS would lose revenue.  
  2. Make sure those firms get onto the framework without having to offer particularly competitive prices, so they will be happy to put senior people onto government work without worrying about the rates.  
  3. Ensure that there are a large number of “SMEs” (smaller firms) who win a place on the MCF. Ministers can then supposedly support the small business agenda and announce that “over 50% of the firms selected are small firms”.
  4. But also make sure there is no need for any government department to actually use any of these small, lower cost firms.

So if these are indeed the objectives, how has CCS given itself the best chance of achieving this?

A Dodgy Price Evaluation

The way price is evaluated is a major factor here. So Lot 1 is general “business”, and up to 75 suppliers will be appointed to this Lot. Here, when the bidders “price” is evaluated, it is weighted at 90% of the total marks available. But the other 10% is just a tick box to say you will deliver the services (which is odd in itself – why would I be bidding otherwise?)

Price is calculated as the median of the prices offered for the 6 grades, from junior consultant up to Partner level. So basically, this is purely a price selection. The cheapest firms, which will be small firms that few of us will have ever heard of, will win a place. And because no-one has heard of them, and (in some cases at least) they are not very good, which is why they are cheap, they won’t be used much. But CCS and Ministers will have lots of SMEs on the list to boast about.

So then how does CCS make sure that the big firms succeed? For Lots 2, 3 and 4, price is only weighted at 10% of the total marks.  The rest come from essay-type questions in which the firms have to show extensive capability. There is plenty of scope for some flexibility in the marking too, and given the low weighting, price barely matters.   I would bet my mortgage that the “usual suspects” will all win places here.  

But just to make sure that those firms don’t have to worry about not making enough money, the price on which marking is based is not calculated as the average (the mean) of the 6 grades, which would seem to be a logical approach, or a weighted average rate based on likely frequency of use of each grade. Instead, it is the average of just two grades, the two “middle” ones (senior consultant and principal consultant / associate director). Actually, that would seem to be the same as the “median” price which is how Lot 1 is defined – it is not clear why different terminology is used.

So that means you don’t have to worry much about the price you put in for Partner. There is one more constraint in that for each grade, the price must be between 10 and 50% lower than the grade above.  But that isn’t too much of a hardship – for instance, you could put in this bid:

FIRM A

Partner                               £6000 a day

Director                              £3000

Principal consultant        £1500

Senior consultant             £1300

Consultant                          £1150

Analyst                                £1000

Your score would be based on the average of £1300 and £1500, so that is £1400, which is probably not too out of line with many bids. But once you win a contract, you can legitimately put your Partners in at £6K a day!

This is an “Illegal” Evaluation Methodology

There is also a technical/legal issue here, in that your evaluation score could be the same as another bid that puts in much lower rates for the top two grades (or indeed the lowest two), as long as you offer the same rates for those two in the middle. That seems to break fundamental rules of public procurement, that you have to make “value” your selection factor and you have to show you have a “fair” process.  So Firm B (below) scores fewer points in the evaluation than firm A, even though their pricing is much better value overall!

FIRM B

Partner                               £2000 a day

Director                              £1750

Principal consultant        £1500

Senior consultant             £1320

Consultant                          £900

Analyst                                £600

I can think of no reason why the average of the 6 grades has not been used – other than to help the big firms charge a fortune for their Partners. Unless I’ve missed something here, it feels like either a real error or there is something odd going on. I’m not a conspiracy theorist, but you do sometimes wonder if there is some sort of plan for certain firms to suck as much money as possible out of the public purse at the moment?   

This is the Argos Catalogue, not a “Framework”

Finally, there is another somewhat technical issue, in that users of the framework who want to choose a supplier for an individual project should (to be legally compliant) in most cases invite all the suppliers listed in the Lot to bid. But if you have to ask 75 firms (Lot 1) or even 30 firms (Lots 2 to 6 ) to put in proposals, that is quite a workload to manage and evaluate.

So I suspect CCS assumes that many users will just choose their favourites from the list, even if this technically breaks the regulations. We’re going back to the old days when I worked in government in the 1990s and that was how frameworks were generally used. Budget holders just picked their favourites from a preferred supplier list. The approach didn’t deliver value for money for the taxpayer then, and it doesn’t now.

But again, having such extensive lists of suppliers ensures that there is plenty of choice on the framework for users, so CCS maximises its own revenues. I’m afraid that looks like a major driver here, along with keeping Ministers, budget holders and the big firms happy.

What Does the Lord Think?

I do also wonder what Lord Agnew, the Cabinet Office Minister, thinks of this, or if he is even aware of what is going on. It was Agnew who wrote to senior civil  servants last September telling them to “rein in spending on consultants” and that Whitehall was being “infantilised” by their over-use. 

But when you see headlines in a year or two about “firms charging £6K a day for consultants”, you know why. Basically, the government, through Crown Commercial Service, has designed its procurement process to allow that. This is all very disappointing, given the undoubted talent of the people in CCS involved in this exercise.

PS  Buying consulting services based on a “day rate” model is almost always the wrong way to do it, anyway. More on that another day.

PPS There is no mention of “social value” in the tender either.

/1 Comment/by