[inductive_magic]

>There’s nothing stopping Jian Yang from creating newpipeline.ai

As someone leading a product team which "only calls apis" in this context: this is a very premature take. Hear me out :-)

Robust LLM-powered software requires

* very thoughtful design of prompt templates

* understanding of top_p and temperature in the context of said templates and their parameter space

* very thoughtful design of representative test cases for a given combination of prompt template and api params. without these, you're not even able to reason about the value range of the function you're developing

* execution and evaluation of those tests

* maintenance of all above

...and that's just talking about ensuring the desired output types in one closed context. I won't go into the creativity required to solve more complex problems (content injections, for one). Let me just say this: I won't lose sleep because anyone could just replicate our applications. The opposite is the case: I invite anyone to try and catch up. Good luck with that.

What you wrote might apply for prototypes of zero-shot applications, but not for production-ready software, letalone production-ready software that solves problems which involve more than one isolated LLM-call.

[infecto]

I don't think either of you are entirely wrong but I think it can be broken down more generally.

There are very few web applications that have any real proprietary implementations that are impossible to replicate. Its a combination of factors that builds the potential moat to the business.

[benob]

I think the point of the parent was that you can build expertise in all of those in a relatively short time frame (esp. when more developers will start building up experience), compared to acquiring a large customer base or a large dataset.

[msp26]

Any tips on building robust LLM software? And yeah I agree on how important test cases are. Having some sort of objective benchmark for judging how effective prompts are is really useful.

[inductive_magic]

>Any tips on building robust LLM software?

Honestly, it's difficult.

One reason why we are very happy with our results is because the person in charge of experimenting with the LLM-flow, while highly intelligent, is not talented with languages (honestly, he's the opposite).

This forces him to come up with creative solutions where others might get more mileage just with flawless prompts. Thanks do this, we discovered some really interesting tricks which help us solve problems that the available literature does not discuss.

Based on the flows he designs, I revise his prompt templates for more precision and token efficiency.

>And yeah I agree on how important test cases are. Having some sort of objective benchmark for judging how effective prompts are is really useful.

Its not just about effectiveness, but about ensuring that no false-positive-inferences make it into production: for a given prompt template, deeply investigate which edge cases of input data would lead to false positives. then, adapt the template and api-params until the unit test has 100% success.

[zelias]

> Thanks do this, we discovered some really interesting tricks which help us solve problems that the available literature does not discuss.

Care to share? ;)

[intended]

Dont.

If you must, there is a continuum of tasks that range from suitable to risky in production settings.

Most definitely choose things on the suitable side of that scale (Eg - text generation, or classification).

More complex tasks like Data to text or Summarization? I personally would always avoid it, except if there are certain very specific workflows for your team/task.

Further, Its not just test cases - its an entire evaluation and prompt versioning layer. Of the few that I am aware of, most are not even openly available (Including Azure Prompt flow)