Why this code doesn't generate circles but instead does this?

Why this code doesn't generate circles

Primary issue:

It looks as though you've ended up with a moiré effect. You've calculated the colour as if it were a single channel with 2²⁴ levels, not three channels each with 2⁸ levels. ... @ Weather Vane

To achieve a grey scale, assigned the values [0...0xFF] to each primary.

// Sawtooth grey scale.
data[iPIXEL_COLS + j]  = trunc(0xFF  cosdis) * 0x010101;

// Simplifications exists.  Verbose for clarity.
// Other color wheel models exist.
unsigned index  = trunc(0xFF*3 * cosdis);
unsigned hue = index%0x100;
unsigned pair = index/0x100;
uint32_t r,g,b;
switch(pair) {
  case 0: r = hue; g = 0; b = 0xFF-hue; break;
  case 1: r = 0xFF-hue; g = hue; b = 0; break;
  case 2: r = 0; g = 0xFF-hue; b = hue; break;
}
r