Finding an operator for 3 distinct values with bitwise operations

Your operator is symmetric, ie

x op y == y op x

Further, A op x == A for any x, otherwise if x == y then x op y == B and x op y == C if x != y.

Hence, the operator can be implemented as

    unsigned op0(unsigned x,unsigned y) {
        if (x == A or y == A) return A;
        if (x == y) return B;
        return C;
    }

---

Once you choose values for A,B,C its just a matter of refactoring the above function to access and set individual bits. Here:

unsigned A = 0b00;
unsigned B = 0b11;
unsigned C = 0b01;

The first bit indicates whether the value is an A. The other bit distinguishes B, and C. Rewriting the above ifs in bit operations is straightforward:

• x op y has the first bit set if x and y have the first bit set.
• x op y has the second bit set if x and y have the first bit set and both have the second bit set or neither of them has it set (ie their second bit is equal).

For readability I use temporaries for the individual bits:

unsigned op(unsigned x,unsigned y) {
    bool x0 = 0b01 & x;
    bool x1 = 0b10 & x;
    bool y0 = 0b01 & y;
    bool y1 = 0b10 & y;    
    return ((x0 and y0)>>0) | ((x1==y1 and x0 and y0)

This answer shows a systematic approach to defining your operator for any arbitrary truth table, using K-Maps and using 2 bits to encode A, B, and C.

We'll (fairly arbitarily) assign:

• A = 00
• B = 01
• C = 10

We can then take our truth table:

And rewrite this in terms of the two bits we assigned to each letter. We use X to mean "Don't care".

Our notation is that, if X = 01 (i.e. it has the value B), then X1 = 0 and X2 = 1.

We can then construct K-maps for the two outputs O1 and O2, in terms of the 4 input bits X1, X2, Y1, and Y2. I'm not going to describe in detail how to do K-maps as that's rather a large topic (and Wikipedia does it pretty well), but I'll work this example so you can see it in practice.

O1:

This is the K-Map for O1 in terms of X1, X2, Y1 and Y2. I've marked the two fairly obvious groupings which describe when O1 is 1 (making use of the "don't care's"):

Here, the rows are values for the X input (so X1X2 = 01 means that X1 = 0 and X2 = 1, etc), and the columns are values for the Y input.

This gives:

O2:

This is the K-Map for O2 in terms of X1, X2, Y1 and Y2. Here the 1's are annoyingly dispirate and hard to group, so I'm going to group the 0's instead:

Then by De Morgan's laws:

---

You can then implement this as e.g.:

auto op (auto x, auto y) {
    auto x1 = x & 0b01;
    auto x2 = (x & 0b10) >> 1;
    auto y1 = y & 0b01;
    auto y2 = (y & 0b10) >> 1;
    auto o1 = (x1 & y2) | (x2 & y1);
    auto o2 = (x1 | x2) & (y1 | y2);
    return (o1 

In complement to Sebastian Redl's answer, it is possible to follow a SWAR approach and vectorize up to 32 operators within a std::uint64t register. The operator becomes:

template
auto op (T x, T y)  {
    T abit  = (x & y) & 0x5555555555555555;
    T amask = (abit 

Basically I come to the same conclusion as https://stackoverflow.com/a/79855949/6607497 does.

With "simplified C" and A == 0b001, B == 0x010, C == 0b100 the table is:

Looking at the ones in the result, the result is A if any of the operands is A. (Else) The Result is B if both operands are either B or C (otherwise the result is C).

So the result would be

o1 == 0b001 || o2 == 0b001 ?
 0b001 :
 o1 == o2 ?
  0b010 :
 0b100