williamtries , Englisch
@williamtries@floss.social avatar

I just posted a short tutorial on setting up a simple on your running . The LLM (7b alpaca in this case) is not terribly accurate but possibly useful in some cases.

Warning! You phone will get hot. I haven't tested it long enough to know if it will harm your device, but do be careful.

Oh! I have a website now! Wow! My history with websites is hit and miss, so enjoy it while it lasts. I have half a dozen posts in the works already.

https://www.williamtries.ovh/llmonpmos/

okias ,
@okias@floss.social avatar

@williamtries I'm happy to see LLM run for you, can you try run it on OpenCL? The GPU was handling GPT-2 much better when I tried (100% 8 cores versus 20% 2 cores + GPU). You'll need additional patch for mesa, can be found here: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/59440

pocketvj ,
@pocketvj@fosstodon.org avatar

@okias @williamtries
how would we apply this?

okias ,
@okias@floss.social avatar

@pocketvj @williamtries using mrtest tool :)

pocketvj ,
@pocketvj@fosstodon.org avatar

@okias @williamtries

how do we find the correct gpu name to enable ?

RUSTICL_ENABLE=sdm845gpucc
or is it:
qcom or adreno ?

karolherbst ,
@karolherbst@chaos.social avatar

@pocketvj @okias @williamtries freedreno or msm

pocketvj ,
@pocketvj@fosstodon.org avatar

@karolherbst @okias @williamtries
perfect, thanks...
still no gpu support on ... guess i am missing some VAAPI hack 🤷

karolherbst ,
@karolherbst@chaos.social avatar

@okias @williamtries maybe we should just merge freedreno support at this point, because it's still not upstream?

  • Alle
  • Abonniert
  • Moderiert
  • Favoriten
  • •
  • random
  • haupteingang
  • Alle Magazine
    •