Mac OS: Bringing metal support

[baileyheading]

Work is underway to get metal working for this project. Keen to get help from anyone interested.

TODO:

• add conv_tranpose_1d for metal (already has cpu implementation)
• modify ggml_pad backend and added ggml_pad_ext for metal
• add pad_reflect_1d to metal (already has cpu implementation)
• add unfold_1d to metal (already has cpu implementation)

Existing PRs from @balisujohn decent example to follow for the changes done to CUDA already:
https://github.com/ggerganov/ggml/pulls/balisujohn

[baileyheading]

that was fast! before it had problem haha

gpt_vocab_init: loading vocab from '../models/tokenizer.json'
gpt_vocab_init: vocab size = 255
autoregressive_model_load: loading model from '../models/ggml-model.bin'
autoregressive_model_load: ggml tensor size    = 432 bytes
autoregressive_model_load: backend buffer size = 1889.29 MB
autoregressive_model_load: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Max
ggml_metal_init: picking default device: Apple M3 Max
ggml_metal_init: using embedded metal library
ggml_metal_init: GPU name:   Apple M3 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 103079.22 MB
autoregressive_model_load: model size  =  1510.54 MB
ggml_metal_free: deallocating
 41984 diffusion_model_load: loading model from '../models/ggml-diffusion-model.bin'
diffusion_model_load: ggml tensor size    = 432 bytes
diffusion_model_load: backend buffer size = 689.28 MB
diffusion_model_load: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Max
ggml_metal_init: picking default device: Apple M3 Max
ggml_metal_init: using embedded metal library
ggml_metal_init: GPU name:   Apple M3 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 103079.22 MB
diffusion_model_load: model size  =   688.49 MB
GGML_ASSERT: /Users/user_name/Documents/GitHub/tortoise.cpp/ggml/src/ggml-metal.m:2216: ne00 % 4 == 0
GGML_ASSERT: /Users/user_name/Documents/GitHub/tortoise.cpp/ggml/src/ggml-metal.m:2216: ne00 % 4 == 0

[baileyheading]

Some lldb (debug) info

GGML_ASSERT: /Users/me/Documents/GitHub/tortoise.cpp/ggml/src/ggml-metal.m:2216: ne00 % 4 == 0
GGML_ASSERT: /Users/me/Documents/GitHub/tortoise.cpp/ggml/src/ggml-metal.m:2216: ne00 % 4 == 0
GGML_ASSERT: /Users/me/Documents/GitHub/tortoise.cpp/ggml/src/ggml-metal.m:2216: ne00 % 4 == 0
GGML_ASSERT: /Users/me/Documents/GitHub/tortoise.cpp/ggml/src/ggml-metal.m:2216: ne00 % 4 == 0
Process 38817 stopped
* thread #5, queue = 'ggml-metal', stop reason = signal SIGABRT
    frame #0: 0x000000018e276a60 libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`:
->  0x18e276a60 <+8>:  b.lo   0x18e276a80               ; <+40>
    0x18e276a64 <+12>: pacibsp 
    0x18e276a68 <+16>: stp    x29, x30, [sp, #-0x10]!
    0x18e276a6c <+20>: mov    x29, sp
Target 0: (tortoise) stopped.
(lldb) tb
No breakpoints currently set.
(lldb) bt
* thread #5, queue = 'ggml-metal', stop reason = signal SIGABRT
  * frame #0: 0x000000018e276a60 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x000000018e2aec20 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x000000018e1bba30 libsystem_c.dylib`abort + 180
    frame #3: 0x0000000100677378 libggml.dylib`__ggml_metal_graph_compute_block_invoke.cold.40 + 88
    frame #4: 0x000000010066287c libggml.dylib`__ggml_metal_graph_compute_block_invoke + 21672
    frame #5: 0x000000018e0fe428 libdispatch.dylib`_dispatch_client_callout2 + 20
    frame #6: 0x000000018e112850 libdispatch.dylib`_dispatch_apply_invoke3 + 336
    frame #7: 0x000000018e0fe3e8 libdispatch.dylib`_dispatch_client_callout + 20
    frame #8: 0x000000018e0ffc68 libdispatch.dylib`_dispatch_once_callout + 32
    frame #9: 0x000000018e1119b8 libdispatch.dylib`_dispatch_apply_redirect_invoke + 260
    frame #10: 0x000000018e0fe3e8 libdispatch.dylib`_dispatch_client_callout + 20
    frame #11: 0x000000018e110080 libdispatch.dylib`_dispatch_root_queue_drain + 864
    frame #12: 0x000000018e1106b8 libdispatch.dylib`_dispatch_worker_thread2 + 156
    frame #13: 0x000000018e2aafd0 libsystem_pthread.dylib`_pthread_wqthread + 228

[baileyheading]

The GGML Assert is from ggml.metal.m I think. I don't know if that causes the abort or something else

EDIT: yep, the asserts cause the block. that means there is probably an issue somewhere, but I can at least proceed to the next stages for now by skipping over it

[baileyheading]

@balisujohn so this is what you were referring to about the unsupported ggml operations? It's very fast at least up until the problems

ggml_metal_graph_compute_block_invoke: error: unsupported op 'UPSCALE'
GGML_ASSERT: /Users/me/Documents/GitHub/tortoise.cpp/ggml/src/ggml-metal.m:918: !"unsupported op"

[baileyheading]

It says it's not supported, but it looks supported to me!

EDIT: right, _EXT is different

[balisujohn]

This is actually quite suspicious, ggml_upscale_ext should use GGML_OP_UPSCALE

[baileyheading]

This is actually quite suspicious, ggml_upscale_ext should use GGML_OP_UPSCALE

any thoughts on where to look?

[baileyheading]

Actually! I'm comparing the CUDA code
CUDA

        case GGML_OP_UPSCALE:
            ggml_cuda_op_upscale(ctx, dst);
            break;

METAL

METAL checks for support with a function to return a bool, but I tried setting it to "true" already with no luck

EDIT:
METAL
Actually it has this

[baileyheading]

I believe that PAD is also supported

ggml_metal_graph_compute_block_invoke: error: unsupported op 'PAD'
GGML_ASSERT: /Users/baileyheading/Documents/GitHub/tortoise.cpp/ggml/src/ggml-metal.m:919: !"unsupported op"
ggml_metal_graph_compute_block_invoke: error: unsupported op 'UPSCALE'
GGML_ASSERT: /Users/baileyheading/Documents/GitHub/tortoise.cpp/ggml/src/ggml-metal.m:919: !"unsupported op"
ggml_metal_graph_compute_block_invoke: error: unsupported op 'PAD'
GGML_ASSERT: /Users/baileyheading/Documents/GitHub/tortoise.cpp/ggml/src/ggml-metal.m:919: !"unsupported op"

Update: I'm removing some of the GGML_ASSERTs to see whether it's just the detection of support that is broken

[baileyheading]

It seems to me like these numbers are somehow wrong for the Metal implementation

        case GGML_OP_UPSCALE:
            return true;
        case 49:
            return true;

I did some printouts and apparently the upscale one it wants is 49, which doesn't seem to be what GGML_OP_UPSCALE is defined as. I added 49 for debugging, but I'm gonna swap that op out for it to try and get by the error

Update: Yeh it got past, but then got error for a different op that also exists, so I'm going to figure there's something wrong with these numbers somewhere

[balisujohn]

Possibly i made a mistake adding other new ops. Though not sure why it would work in cuda and not metal.

[baileyheading]

@balisujohn they're out by two.. which seems consistent with this kind of thing

dst->op in default case: 49 (what was sent from main.cpp presumably)
dst->op in default case: 51 (Metals definition of the number)
ggml_metal_graph_compute_block_invoke: error: unsupported op 'UPSCALE'

but if they're only defined in one place, I don't understand how this would happen

[baileyheading]

I numbered them. What confuses me is that it believes 51 is upscale in metal but the incoming operation thing says it is 49 and it has the name "UPSCALE". where are they defined differently?

[balisujohn]

These are out of order with the OP enum, possibly leading to at a minimum the wrong op names being printed. All of the things I linked should reference the same ops in the same order.

https://github.com/balisujohn/ggml/blob/e9ac38f82c0956cf6f4e86206d4503095c25d47c/src/ggml.c#L2701
https://github.com/balisujohn/ggml/blob/e9ac38f82c0956cf6f4e86206d4503095c25d47c/include/ggml/ggml.h#L424
https://github.com/balisujohn/ggml/blob/e9ac38f82c0956cf6f4e86206d4503095c25d47c/src/ggml.c#L2613

[balisujohn]

(I think the op failing on metal actually isn't upscale)