Which of the following would the security engineer set as the subnet mask for the servers below to utilize host addresses on separate broadcast domains?

Each / refers to the mask length.

We only care about our final octet, so the mask will need to be *at least* 24 in length.

Why?

Because 255.255.255.X is at least 24 ones. Count them:

255 = 1111 1111

3 each 255’s = 8*3 = 24

The question is asking us to consider what kind of subnet mask will force all 3 of those IP addresses to be on different subnets. Different subnets mean different broadcast domains, which means less overall traffic for the switch/router. This is why subnet masking is A Good Thing.

A subnet mask of 255.255.255.248, or /29, looks like the following:
1111 1111 | 1111 1111 | 1111 1111 | 1111 1XXX

Which will force subnets the 3 IP addresses, by the final octet, to necessarily be on different subnets. I suppose that /30 would do the same trick, but this is the best I can figure out to understand the answer’s rationale.

0000 0110 – 6
0000 1001 – 9
0001 0100 – 20

A subnet mask smaller than /29 would put at least .6 and .9 on the same subnet. /29 is just the right size.