[walrus01]

If and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era.

edit: There's already a number of LLM which are intended for outgoing data loss protection to redact or prevent PII from escaping. Is anyone specifically working on a training set and agent that is specialized in reviewing "is this legit to pay", as a sub-task or filtering step in an AP workflow? I suppose it's a GIGO problem, as it would work best only if you have suppliers enrolled in some kind of existing db, with a specific contracted format for invoices, and correlating with project numbers/cost codes.

[specproc]

I think it would be absolutely insane to hand over a serious-sized company's books to an LLM.

As a small consultancy though, looking forward to my next filing, and having just moved to a new and better-specced jurisdiction, I'm sorely tempted to outsource to Claude.

I've had mixed experience with accountants in the past. No horror stories, but I often feel I'm not getting everything laid out clearly, and that I don't fully understand the process.

I've got plenty of reasons to dislike LLMs in my own work, but when dealing with well-scoped but professionally gatekept things like tax or property transactions, they're an absolute godsend.

[order-matters]

i specialize in invoice related analytics and business processes. AI can be useful for data extraction and saves me some typing time. even so, i dont blindl trust it because it sometimes makes very reasonable mistakes because invoices, quotes, and POs are sometimes structured in very informal ways. people misuse the lines, sublines, totals, and other data fields. they are often technically incorrect but when you look at them you know what they mean. i am not sure how to hand that off to something that has plausible deniability to guess even if it doesnt know

sometimes details are just in notes at the bottom and are applied to selectively applied to line items. sometimes the charge doesnt exist anywhere on the bill but there is an understanding (due to a separate agreement) of additional charges to be paid as a result of the invoice.

taxes sometimes are or are not explicitly stated

tariffs sometimes are or are not explicitly stated

when things are not explicitly stated or line item'd, they will usually still appear in the invoice total. so you have item 1 - $500, item 2 $500. total: $1300

At the end of the day invoices are often part of an ongoing communication / conversation between two organizations and they are created with an assumption that a rational and reasonable human who is in the loop with that conversation is going to read it.

[mediaman]

You can fix this simply by using normal controls.

That's why we have purchase orders that can only be entered by buyers. Product is received and approved by buyer. Invoice goes to accounting, who can't approve it unless there's a matching purchase order and receiver.

Yes, letting agents do whatever they want leads to disaster. But humans are gullible stochastic token generators as well. And that's why the problem is already solved.

[walrus01]

Indeed so, a fairly mundane RFP, RFQ, buyer, receiver, accounts payable process will stop a lot of problems. If an agent is inserted at some stage in the process with a clear path to make a ticket/escalate to a human if it sees something it doesn't understand, the risk isn't absurdly high, in my opinion.

I've seen so many reports of humans with the authority/ability to execute an outgoing SWIFT transfer who've been social engineered into sending money to fraudsters... Or even just the basic low level "Hey I'm your boss sending you an SMS, please go buy some gift cards and scratch them off and send me the codes". No AI involved whatsoever.

The danger exists where some true believer AI evangelist type of management person tries to fully automate the entire purchasing and AP workflow, which I'm sure some people will attempt soon, with varying degrees of success.

[mediaman]

I have seen the SWIFT thing happen for $100k. I think AI could actually be better for this, because it's often easier to implement hard rules for the AI.

With the SWIFT incident I saw, there was a rule that no payment can go to a vendor's bank that isn't a current, approved vendor. But the rule was not enforced in software: it was an internal accounting rule that humans were supposed to follow. The AP person "thought it had been approved" because there was a similar transaction with a different company that was a new vendor at a similar time. The other transaction was legitimate, the fraudulent spoofer wasn't. The wire got sent to a party in China.

With AI agents, if you approach it from the perspective that it will be gullible and trickable by fraudsters, you build in these hard guardrails. With humans, it's much easier to believe that "we trained Lucy on this procedure" will work in all circumstances, even if Lucy still has the technical ability to bypass the official procedure.

In these cases, it starts looking a lot more like traditional software, with your little AI chaos monkeys constrained in little boxes within the software chain.

[walrus01]

Another thing that seems to be disturbingly common these days is some party involved in residential real estate transactions having a RAT on one or more PCs, and/or compromised email account, intercepting an email at the closing stages of a house purchase and sending someone information to send a wire transfer to a fraudulent location.

[jimnotgym]

But the hard rules only work up to the point that there is an exception. See my other post above. Occasionally a very senior person (hopefully senior) has to approve a payment that is outside of the rules, because it is something the rules did not anticipate (and now they are hardcoded into software and can't be changed).

[jimnotgym]

I run a team that includes people that do this kind of work using ocr software that matches invoices to po's. No AI needed. This is a solved problem. Why are there people involved? Because sometimes the invoice and po don't match. For instance, price on invoice is higher than po, refer to buyer. Buyer is sick, supplier puts you on hold, no parts for your factory, lose millions... Would you trust an AI to choose what to do next? This might get referred to me to resolve and make a decision, not just on the facts available, but on other facts I can discover, and years of experience. I might end up making an unauthorised payment, would you give an AI that power?

[dylan604]

Sounds like Enron was just a bit early. They could have just blamed the AI system instead of becoming a meme with "Enron math"

[charcircuit]

With AI you can scale the protection against social engineering. Where with humans you have to start from scratch each time and they are more likely to mess up.