[badsectoracula]

The information about finetuning is interesting (it is something i'd like to do myself at some point, though i'll wait until i can do it with local hardware :-P). However FWIW LLMs are generally good at following a specific style when given examples.

As an example i asked Devstral Small 2 to write some docs for my LIL scripting language in the following style (this is copied from the DirectDraw documentation, edited to be text friendly):

    IDirectDraw7::CreateClipper
    ---------------------------

    The IDirectDraw7::CreateClipper method creates a DirectDrawClipper object. 

        HRESULT CreateClipper(
        DWORD dwFlags,
        LPDIRECTDRAWCLIPPER FAR *lplpDDClipper,
        IUnknown FAR *pUnkOuter
        );

    Parameters

    * dwFlags - Currently not used and must be set to 0. 
    
    * lplpDDClipper - Address of a variable to be set to a valid
        IDirectDrawClipper interface pointer if the call succeeds.
        
    * pUnkOuter - Allows for future compatibility with COM aggregation features.
        Presently, however, this method returns an error if this parameter is
        anything but NULL. 
        
    Return Values

    If the method succeeds, the return value is DD_OK.

    If it fails, the method can return one of the following error values: 

    * DDERR_INVALIDOBJECT
    * DDERR_INVALIDPARAMS
    * DDERR_NOCOOPERATIVELEVELSET
    * DDERR_OUTOFMEMORY

    Remarks

    The DirectDrawClipper object can be attached to a DirectDrawSurface and used
    during IDirectDrawSurface7::Blt, IDirectDrawSurface7::BltBatch, and
    IDirectDrawSurface7::UpdateOverlay operations.

    To create a DirectDrawClipper object that is not owned by a specific
    DirectDraw object, use the DirectDrawCreateClipper function.

    Requirements 

    Windows NT/2000: Requires Windows 2000.
    Windows 95/98: Requires Windows 98.
    Header: Declared in ddraw.h. 

    See Also

    IDirectDrawSurface7::GetClipper, IDirectDrawSurface7::SetClipper

And it did a fine job. I put the full transcript in[0] to check out. The neat bit is that it can even handle weird formats like a custom documentation format i have (which only exists in my PC because i haven't released it anywhere) for a "master document" that can then be converted to various other file types. I gave it an example of some code in that and asked it to convert the documentation to it (this is part of the transcript at the end). Then i copy/pasted the generated code to a new file (adding a few extra lines the doc system expects which weren't part of the example - BTW i did not had to modify the generated code at all) and from that i generated a CHM file[1]. FWIW here is a comparison with the DirectX page i copied[2] (though consider that the generated pages went through the doc format which forces its own style and the textual output in the transcript matches the given style better).

[0] https://app.filen.io/#/d/9f4c1225-3527-4f16-a522-0678342120c...

[1] http://runtimeterror.com/pages/iv/images/45f8df428afe4fe6b6a...

[2] http://runtimeterror.com/pages/iv/images/ee58032790a049d7e74...

[theletterf]

Interesting! I think the advantage of style fine-tuning is that you might not have to provides that much context upfront. Also, it's kind of magical to have an LLM just do something out of the box. I'll compare my local fine-tuned models against the baseline with instructions and see how they fare.

[badsectoracula]

Yes, TBH one of the reasons i want to try and finetune my own is to teach it stuff that now i have to explain before anything can be done :-P.

Unfortunately i only have a 24GB GPU - and an AMD one at that - so there isn't much i can do on that front. Supposedly a 24GB GPU is enough for finetuning a 24B model with 4bit QLoRA, though when i tried it with some finetuning app (in an official docker container) it barfed at Mistral's weird template or something and i lost interest after that.

[theletterf]

Try Runpod or similar services: you can fine-tune stuff for the price of a latte. Stanford's NLP course recommends them: https://cs336.stanford.edu/

[badsectoracula]

I've heard about this but i prefer to avoid anything cloud based (not just for AI but in general). I try to avoid relying on stuff i have little to no control over.

[zahlman]

> However FWIW LLMs are generally good at following a specific style when given examples.

In your experience, is it worthwhile to have an agent create a "skill" for itself for following the style? Or is it a better use of context to just have it review the examples?

[badsectoracula]

TBH i actually haven't tried that at all (or "skills" in general since my PC isn't fast enough for doing much more than chatbot-style interactions with maybe the occasional -custom- MCP server to let it access files, external data, etc -- i'd love to get a second PC that i can use for LLMs, among other stuff, but that's far in the future).

Out of curiosity i just tried having the LLM generate a document describing the style, then (after a memory reset) i asked it to use the document and it did more or less the same job, however the document also had an example and the example seemed to do most of the "real job" of describing the style because at first the function docs that were generated were prefixed with 4 spaces - like the example in the doc (but the example in the doc had four spaces for indentation) and after i edited the doc to remove the spaces the function docs it generated were like those generated in the transcript i linked above.

Which makes me think that the example is the better approach (since the doc also had an example) and perhaps the best is to give an example with clarifications about the parts the example doesn't cover (like using a 80 line character limit).

FWIW when i wanted the LLM to write some new C source + header files some time ago, i also pointed it to an existing pair of C/H files to use for the code style and it worked fine, so at least in my experience examples seem to work very good.