sha256sum Illegal instruction on musl amd64

[Denys Vlasenko]

On Wed, Mar 1, 2023 at 12:01 PM alice <alice at ayaya.dev> wrote:
> On Tue Feb 28, 2023 at 11:17 PM CET,  wrote:
> > I'm having an intermittent issue with "BusyBox v1.36.0 (2023-01-03 22:49:12
> > UTC)" (the one from the Docker image busybox:musl) when running on amd64 GitHub
> > actions runner VMs (azure).
> >
> > When I use sha256sum it is getting terminated with SIGILL, Illegal
> > instruction. The issue is hard to reproduce but I have a GitHub actions CI/CD
> > job that I can re-run repeatedly (no changes to code, environment, data input,
> > etc) that will occasionally have the issue. I managed to capture a core dump.
>
> this was also reported in http://lists.busybox.net/pipermail/busybox/2023-January/090113.html
>
> it's caused by having a cpu with AVX512 (the github runners do) but not sha_ni,
> and the code that checks it is broken and misdetects sha_ni support when avx512
> exists. the github runners don't have sha_ni, so it breaks exactly there.

The code is:

                if (!shaNI) {
                        unsigned eax = 7, ebx = ebx, ecx = 0, edx = edx;
                        cpuid(&eax, &ebx, &ecx, &edx);
                        shaNI = ((ebx >> 29) << 1) - 1;
                }
                if (shaNI > 0)
                        ctx->process_block = sha1_process_block64_shaNI;

If ebx's bit 29 is set, then shaNI = 1. If it is clear, then shaNI = -1.

I checked
"Intel® 64 and IA-32 Architectures Software Developer’s Manual
Combined Volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D and 4"

to verify that CPUID(EAX=7, ECX=0) and bit position (EBX.29)
is the correct check for SHA-NI.

I double-checked it in Linux kernel source.

I checked disassembly and it correctly sets up registers for CPUID.

Can someone tell me on what CPU this does not work?