hostapd: 4-way handshake and replay counter handling?

[Helmut Schaa]

Hi,

I can sometimes reproduce a 4-way handshake failure with an
Apple iPhone STA and hostapd as authenticator. Under special
circumstances the iPhone just ignores message 3/4 and thus the
4-way handshake times out.

The message exchange looks like this (I can also provide the pcap
if anyone is interested, just need to trim it first):

AP <----> STA

---> 1/4 replay counter 1 nonce A
<--- ACK

<--- 2/4 replay counter 1 nonce B
<--- 2/4 replay counter 1 nonce B (retry)
<--- 2/4 replay counter 1 nonce B (retry)
<--- 2/4 replay counter 1 nonce B (retry)
<--- 2/4 replay counter 1 nonce B (retry)

---> 1/4 replay counter 2 nonce A
<--- ACK

<--- 2/4 replay counter 1 nonce B (retry)
---> ACK

<--- 2/4 replay counter 2 nonce C
<--- 2/4 replay counter 2 nonce C (retry)
<--- 2/4 replay counter 2 nonce C (retry)
<--- 2/4 replay counter 2 nonce C (retry)
---> ACK

---> 3/4 replay counter 3 nonce A
<--- ACK

---> 3/4 replay counter 4 nonce A
<--- ACK

---> 3/4 replay counter 5 nonce A
<--- ACK

---> 3/4 replay counter 6 nonce A
<--- ACK

Here's the according hostapd log:

2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 IEEE 802.11: authentication OK (open system)
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 MLME:
MLME-AUTHENTICATE.indication(11:11:11:11:11:11, OPEN_SYSTEM)
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 MLME: MLME-DELETEKEYS.request(11:11:11:11:11:11)
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 IEEE 802.11: authenticated
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 IEEE 802.11: association OK (aid 1)
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 IEEE 802.11: associated (aid 1)
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 MLME: MLME-ASSOCIATE.indication(11:11:11:11:11:11)
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 MLME: MLME-DELETEKEYS.request(11:11:11:11:11:11)
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: event 1 notification
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: start authentication
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 IEEE 802.1X: unauthorizing port
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: sending 1/4 msg of 4-Way Handshake
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: EAPOL-Key timeout
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: sending 1/4 msg of 4-Way Handshake
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: received EAPOL-Key frame (2/4 Pairwise)
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: sending 3/4 msg of 4-Way Handshake
2011:10:07-13:42:14 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: received EAPOL-Key 2/4 Pairwise with unexpected
replay counter
2011:10:07-13:42:15 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: EAPOL-Key timeout
2011:10:07-13:42:15 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: sending 3/4 msg of 4-Way Handshake
2011:10:07-13:42:16 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: EAPOL-Key timeout
2011:10:07-13:42:16 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: sending 3/4 msg of 4-Way Handshake
2011:10:07-13:42:17 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: EAPOL-Key timeout
2011:10:07-13:42:17 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: sending 3/4 msg of 4-Way Handshake
2011:10:07-13:42:18 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 WPA: EAPOL-Key timeout
2011:10:07-13:42:18 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 IEEE 802.1X: unauthorizing port
2011:10:07-13:42:18 192.168.1.101 hostapd: wlan0: STA
11:11:11:11:11:11 IEEE 802.11: deauthenticated due to local deauth
request

So, the iPhone acks the 3/4-messages just fine but ignores them
for whatever reason. The order is a bit strange due to the unusual
retry of the msg 2/4 which was triggered by loads of traffic on the
channel.

So, in short, hostapd used the first msg 2/4 it received from the iPhone
while the iPhone expected us to use the second msg 2/4 which was the
reply to our second msg 1/4. Since the iPhone used a different nonce
for the second msg 2/4 that might explain why it is rejecting the msg 3/4.
"On reception of Message 2, the Authenticator checks that the key replay
counter corresponds to the outstanding Message 1. If not, it silently discards
the message."

Hence, shouldn't hostapd just discard the first msg 2/4 it receives
from the STA?

As far as I could see this behavior was introduced in commit
22a299ee9d192d06c235428d017234539fbf6a62 ("Improve EAPOL-Key
handshake stability with retransmitted frames").

Thanks,
Helmut

[Jouni Malinen]

Post by Helmut Schaa
I can sometimes reproduce a 4-way handshake failure with an
Apple iPhone STA and hostapd as authenticator. Under special
circumstances the iPhone just ignores message 3/4 and thus the
4-way handshake times out.

Is this with RSN (WPA2) or WPA? How easily can you reproduce this? Does
iPhone manage to connect to the network automatically after this failure
or is some user action needed to recover?

Post by Helmut Schaa
The message exchange looks like this (I can also provide the pcap

I would appreciate seeing a capture file for this type of issues. This
is an unfortunately complex area to handle since there are so many
differently broken implementations out there..

Post by Helmut Schaa
So, the iPhone acks the 3/4-messages just fine but ignores them
for whatever reason. The order is a bit strange due to the unusual
retry of the msg 2/4 which was triggered by loads of traffic on the
channel.
So, in short, hostapd used the first msg 2/4 it received from the iPhone
while the iPhone expected us to use the second msg 2/4 which was the
reply to our second msg 1/4. Since the iPhone used a different nonce
for the second msg 2/4 that might explain why it is rejecting the msg 3/4.

Yes, if the authenticator and supplicant end up using different SNonce,
message 3/4 will be dropped due to mismatch in the MIC. wpa_supplicant
tries to avoid this by not changing SNonce within a connection attempt
(i.e., it gets changed only on reassociation or after message 3/4 has
been accepted). I think this would be the best approach to use on a
station, but well, obviously hostapd needs to try its best even if the
station does not do that.

Post by Helmut Schaa
"On reception of Message 2, the Authenticator checks that the key replay
counter corresponds to the outstanding Message 1. If not, it silently discards
the message."
Hence, shouldn't hostapd just discard the first msg 2/4 it receives
from the STA?

Well, yes, in theory.. However, this is problematic because doing so can
break interoperability with some deployed stations.

Post by Helmut Schaa
As far as I could see this behavior was introduced in commit
22a299ee9d192d06c235428d017234539fbf6a62 ("Improve EAPOL-Key
handshake stability with retransmitted frames").

Some high load cases require more time to be allowed for the supplicant
to reply to the 4-way handshake messages. This change tries to work
around those situations. The commit log seems to be talking mainly about
group key updates, so it could be possible to have different behavior
for the initial 4-way handshake. However, I'm a bit hesitant on changing
anything in this area because that result in re-introducing issues seen
with other stations in the past.. I.e., this is likely to end up being a
compromise that tries to avoid worst issues.

[Helmut Schaa]

Post by Jouni Malinen

Post by Helmut Schaa
I can sometimes reproduce a 4-way handshake failure with an
Apple iPhone STA and hostapd as authenticator. Under special
circumstances the iPhone just ignores message 3/4 and thus the
4-way handshake times out.

Is this with RSN (WPA2) or WPA?

RSN.

Post by Jouni Malinen
How easily can you reproduce this?

Reproducibility is low, maybe one out of 30 tries while constantly generating
traffic on the channel with a second client.

Post by Jouni Malinen
Does
iPhone manage to connect to the network automatically after this failure
or is some user action needed to recover?

The iPhone had a different network configured during that time and
connected to that one instead. Not sure how it would have behaved if
there wasn't a different network around.

Post by Jouni Malinen

Post by Helmut Schaa
The message exchange looks like this (I can also provide the pcap

I would appreciate seeing a capture file for this type of issues. This
is an unfortunately complex area to handle since there are so many
differently broken implementations out there..

See private mail.

Post by Jouni Malinen

Post by Helmut Schaa
So, the iPhone acks the 3/4-messages just fine but ignores them
for whatever reason. The order is a bit strange due to the unusual
retry of the msg 2/4 which was triggered by loads of traffic on the
channel.
So, in short, hostapd used the first msg 2/4 it received from the iPhone
while the iPhone expected us to use the second msg 2/4 which was the
reply to our second msg 1/4. Since the iPhone used a different nonce
for the second msg 2/4 that might explain why it is rejecting the msg 3/4.

Yes, if the authenticator and supplicant end up using different SNonce,
message 3/4 will be dropped due to mismatch in the MIC. wpa_supplicant
tries to avoid this by not changing SNonce within a connection attempt
(i.e., it gets changed only on reassociation or after message 3/4 has
been accepted). I think this would be the best approach to use on a
station, but well, obviously hostapd needs to try its best even if the
station does not do that.

Post by Helmut Schaa
"On reception of Message 2, the Authenticator checks that the key replay
counter corresponds to the outstanding Message 1. If not, it silently discards
the message."
Hence, shouldn't hostapd just discard the first msg 2/4 it receives
from the STA?

Well, yes, in theory.. However, this is problematic because doing so can
break interoperability with some deployed stations.

Hmm, ok, sounds like there are lots of strange client implementations out
there :)

Post by Jouni Malinen

Post by Helmut Schaa
As far as I could see this behavior was introduced in commit
22a299ee9d192d06c235428d017234539fbf6a62 ("Improve EAPOL-Key
handshake stability with retransmitted frames").

Some high load cases require more time to be allowed for the supplicant
to reply to the 4-way handshake messages. This change tries to work
around those situations. The commit log seems to be talking mainly about
group key updates, so it could be possible to have different behavior
for the initial 4-way handshake. However, I'm a bit hesitant on changing
anything in this area because that result in re-introducing issues seen
with other stations in the past.. I.e., this is likely to end up being a
compromise that tries to avoid worst issues.

Understood. And I just realized I didn't have "Work around SNonce updates
on EAPOL-Key 1/4 retransmission" applied. The longer timeout might have
helped in this case too I guess. So, I'll try with that patch applied again ...

Thanks so far,
Helmut

[Helmut Schaa]

Post by Jouni Malinen

Post by Helmut Schaa
"On reception of Message 2, the Authenticator checks that the key replay
counter corresponds to the outstanding Message 1. If not, it silently discards
the message."
Hence, shouldn't hostapd just discard the first msg 2/4 it receives
from the STA?

Well, yes, in theory.. However, this is problematic because doing so can
break interoperability with some deployed stations.

Agreed, what about the following:

Assume we receive a 2/4 reply to our first 1/4 msg and we start sending out
3/4. But if we receive a "different" 2/4 reply afterwards we should maybe
send the next 3/4 retry based on the latest 2/4?

Helmut