[squidfood]

Federal Gov employee here. Can't speak for a project with this scope, but the procurement middlemen get into everything, far for the worse.

Two years ago our team wanted to buy a small cluster (~300 cores, ~$50K). We talked directly to two good vendors (good recommendations from university partners) and came up with a fine machine and 2 bids for it. Sent recommendations to procurement.

Procurement put it out for bid, and a fly-by-night company undercut the bid by $10K... by noticing that procurement had not specified details of service level (that were in the bids we'd gotten and forwarded). Procurement, once it goes there, is a true black box. No communication, no understanding.

Five months later, we were basically delivered 2 pallets of unassembled parts and no instructions. Believe me, we spent 3-4x as much in labor as the $10K savings to get it working, and it's been plagued with issues that would have been under the onsite service warranties for the better companies.

The biggest irony is: I firmly believe that procurement acts this way not because the government is fundamentally incompetent, but because the Public, and thus Congress, BELIEVES we are incompetent, so puts so many levels of "check" bureaucracy in place that the people who know what they want can't participate directly in getting it.

[apendleton]

I just had a conversation with a federal employee last week talking about the issue of procurement offices being a blackbox and procurement officers not understanding the technical specifics of what they're bidding out. His solution was just to become certified as a procurement officer (apparently it's 72 hours of coursework) and deal with the bids himself. He said these kinds of problems have pretty much disappeared for him since then.

[VladRussian2]

> His solution was just to become certified as a procurement officer (apparently it's 72 hours of coursework) and deal with the bids himself. He said these kinds of problems have pretty much disappeared for him since then.

sounds like a loophole one can procure a truck through :)

[allochthon]

According to US federal procurementology, would it be a conflict of interest to be both procurer and recipient of what is purchased?

[apendleton]

He's overseeing the bidding process for contractors for projects that he will supervise; he doesn't personally benefit from the outcome of the selection (i.e., he doesn't own stock in companies that are bidding or anything like that), so I'm not sure how it would be a conflict.

So no, I don't think so.

[nostromo]

> I firmly believe that procurement acts this way not because the government is fundamentally incompetent, but because the Public, and thus Congress, BELIEVES we are incompetent

There's almost 3 million federal workers. Many more if you include people who work on government contracts. The Federal Government is by far the biggest enterprise in the US by both employees and revenue.

With such a large organization, there are undoubtedly large swaths of both incompetence and competence.

The challenge with any large organization is that the rules are there to reign in the bad people, but are equally enforced on the good. (For example, most people won't abuse their company's T&E policy, but some will, so everyone has to be treated as suspect.)

Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us."

[squidfood]

> Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us."

Didn't mean to imply that any sufficiently large organization shouldn't have an audit trail and reasonable accountability!

[mwsherman]

I would offer that those two concepts are optimized for ex-post-facto blame. It’s not simply gov’t, as you point out, it’s large organizations.

To contrast this with more modern techniques, ‘audit trail’ is simply source control. And most of us don’t go into source control looking for a smoking gun.

It’s rarely necessary if a process is agile/iterative. Bugs will be (relatively) small and recent in time. So the notion of going back six months and figuring out ‘what went wrong’ is just not a thing. Wrong happens every day, in small amounts, transparently.

Conversely, a bug if truly large and undetected, and explodes a year from its creation, then the whole team is to blame. We’ve all looked at the code hundreds of times in that period.

[6d0debc071]

> Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us."

Maybe not, but "treat me as a liar" doesn't seem viable either.

Any time you take an extra measurement you introduce a chance for that measurement to be in error. If you make that it so that any single measurement is a show-stopper, then every time you add an extra check you make things a little bit worse, right up to the point where your false positive rate overwhelms your data.

Even with an extremely low error rate, the number of best deals in the world for a given thing is 1. If the goal of your system is to get that, then making any measuring system a single point of failure that tests once is a death sentence. If the number of good and honest software houses that will respond to your call is low, then your false positive rate is going to be extremely high... and again, having such a system is going to be ill-advised.

Especially if the system itself suffers from not having the people who are actually going to be using the system, and people who understand how the systems should be created and run, making at least part of the decisions.

If there were a greater degree of feedback between procurement, requesters, and providers, with the ability to modify the plan, then you could potentially check your work - reducing the consequences of such failures. Not "absolute trust," but at least "hear my side of the story, maybe you've just misunderstood something."

[allochthon]

> Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us."

I agree -- this is not a simple challenge. But I don't think stultifying bureaucracy is the answer, either. There must be some government out there, somewhere in the world, that has sorted out an efficient, effective procurement process.

[saraid216]

> But I don't think stultifying bureaucracy is the answer, either.

Something that seems hard for people to grasp is that "stultifying bureaucracy" wasn't an "answer". It's the natural consequence of not having an answer. We have the luxury of sitting back on an Internet forum pontificating on the drudgery; they have to enact the laws that Congress passed.

> There must be some government out there, somewhere in the world, that has sorted out an efficient, effective procurement process.

The main examples of the same scale and scope as the US are Brazil, China, Russia, and India. All of these have been regularly painted as worse: more corrupt, more stonewalling, more favoritism. Whether that's just American propaganda or if there are specific processes that can be imported without breaking things is worth examining.

[eru]

Perhaps you should look at the smaller countries. Like Singapore?

[allochthon]

I like that idea. There might be a good process in place at a smaller scale that can be scaled up.

What is it about Singapore's procurement process that stands out to you? (I don't know anything about it.)

[eru]

I don't know enough about the process to comment, but judging by the results it must be pretty good. They have a good track record of finishing government projects on schedule and budget.

[dllthomas]

'Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us."'

This might be a good starting point:

http://www.amazon.com/Liars-Outliers-Enabling-Society-Thrive...

[joshdick]

Procurement rules are more to guard against corruption and cronyism than incompetence.

Without them, you could reward your friends with fat government contracts, regardless of what's in the public's interest.

[apendleton]

The irony is that they don't really do that. It's common in both procurement and hiring in the federal government to see bizarrely-specific requirements whose fairly transparent intention is to limit the potential bidder or applicant pool to a single company or individual.

[brianpgordon]

* Must have a company name which SHA-1 hashes to e8b06511aa36381fc2306eb6f8181204585c5453.

Hey, there's a theoretically infinite number of company names which meet this requirement...

[mcone]

> Procurement rules are more to guard against corruption and cronyism than incompetence.

Former federal IT contractor here.

The procurement rules were designed for that, yes. But in real world scenarios, those rules effectively do exactly the opposite. Since there are so many hoops for potential vendors to jump through, only the most established players get to bid on most contracts. And in my experience corruption and cronyism is still alive and well in federal IT contracting.

[squidfood]

Yes. I should have added that I understand the need for basic checks and auditing (and that the public expects us to be uniformly fraudulent as well as incompetent), but that the level of checks and removal from the process of experts who know requirements goes far beyond this need.

It's really all about optics rather than reasonable checks - no department wants to be the one that Congress targets, so especially in poisonous political environments the "checks" are significantly more expensive IMO than the actual "waste, fraud, and abuse" they guard against.

[DannyBee]

"Without them, you could reward your friends with fat government contracts, regardless of what's in the public's interest. "

Except this happens anyway. Much like patents, people just become better at drafting. What is currently done is not an effective mechanism for stopping cronyism and corruption at all. In fact makes it easier in a lot of cases, because it provides plausible deniability (It's not that we gave it to our friend, it's that you didn't meet the requirements!)

[phy6]

And like most things with government, you end up getting both: the bureaucracy to prevent cronyism as well as the cronyism. Speaking from experience. Sure it might be more obvious if the bureaucracy wasn't in place, but it's still there.

[marcosdumay]

Well, federal government worker here too, but not on the US government.

Those rules are there because a malicious worker can cause a huge amount of damage. They are a pain (one entity I worked once spent about $15k (in people time) contracting $100 worth of ssl certificates, in a process that took more than a year (so, no certificates for the site during a period), we were forbiden from contracting the service for more than a year... and the contractor was another governmental entity. The rules are maddening, but they are necessary for a democracy.

The problem is that governmental IT is out of place. The government will never be competent in contracting software development - the only known tool that works in keeping government contracts honest is auctioning, and agile is simply not compatible with auctioning. The only possible way out is by doing IT in-house.

[narag]

An in-house team serving several departments can work well: providing infrastructure, defining communication protocols, quality and security standars, helping subcontracting.

[mbesto]

Gov't procurement is actually no different from any other bureaucratic procurement organization in Big Co. They are always seen as a blocker (rarely an enabler) and the people who sit in their roles usually hate their responsibilities. (I'm generalizing of course, but you get the point) The consulting company I used to work implemented techniques for procurement orgs based mainly on behaviors.[1] It's the first time I've seen such a thing and believe it is something very necessary to scaling procurement organizations.

[1]-http://www.youtube.com/watch?v=7Xgh-A1ZCfg

[bane]

The federal procurement process is one of those cases where separation of concerns hurts things. I know that things are this way to try and prevent bribery and cronyism etc. But it's clear that the pendulum has swung too far.

I've been part of the procurement process a few times from the vendor side and the layers of nested black boxes makes solving procurement issues virtually impossible and once the procurement is made massive overruns are almost inevitable.

[dllthomas]

"But it's clear that the pendulum has swung too far."

While I think that's likely, I'm a little bit hesitant to claim things are clear when we can't observe the alternative.

[jasiek]

Yup. The heavy equipment process create an additional barrier to entry. If you want to bid for a govt contract, you need to hire someone who has done it before, otherwise you'll drown in paperwork. What's really sad about this situation is that it is stable - eg. existing stakeholders (contractors & govt procurement personnel) will work hand in hand to ensure that the process stays in place - the former to make more money and the latter to retain their jobs.

[emiliobumachar]

Out of curiosity, why aren't you naming the no-support supplier? Are there any rules forbidding you to do so?

[squidfood]

Honestly, tiny company, couple years ago, don't remember now, I wouldn't be surprised if it was a couple dudes who look at Fed announcements and say "hey, I can buy that off the shelf and ship it to the feds for a profit".

[batgaijin]

maybe watson should handle procurment

[allochthon]

This is distressing.