How to build GRE over IPsec on AWS Cisco CSR1000v
Introduction
GRE tunnel is an easy way to provide point-to-point connection for two sites to talk to each other.
However, GRE does not provide any encryption, so it’s often paired with IPsec to make sure any sensitive data is encrypted and only the authenticated peer is able to decrypt it.
GRE over IPsec refers to the mechanism that packets are first encapsulated in GRE, and then using IPsec for encryption.
While GRE over IPsec tunnel is not complicated to set up between on-premise network routers, a cloud environment may pose a challenge that hinders the tunnel establishment.
This post will explain where is the problem and what is the solution.
For a complete configuration example, please check this article “Cisco: GRE Tunnel + NAT + IPsec“.
IPsec mode
IPsec has “tunnel mode” and “transport mode”. Tunnel mode is the default.
In Tunnel mode, the entire original IP packet is encapsulated to become the payload of a new IP packet. Additionally, a new IP header is added on top of the original IP packet. This new IP header(see New IP Hdr in purple) includes the IPs of tunnel endpoints, i.e. the IPs of the gateways. This allows tunnel mode to protect against traffic analysis, since attackers can only determine the tunnel endpoints.
Transport mode retains the original IP header(see IP Hdr in green), there is no new IP packet created. It will only encrypt the data itself.
GRE over IPsec
GRE encapsulation adds a “GRE” flag and “GRE IP Hdr” on top of the original IP packet. The “GRE IP Hdr”(in pink) includes the source/dest address of the tunnel.
In “GRE over IPsec”, IPsec is applied after GRE encapsulation.
As Figure 2 shows:
- in tunnel mode: “GRE IP Hdr + Original IP packet” is encrypted.
- in transport mode: only Original IP packet is encrypted. The “New IP Hdr”(in purple) is actually the same as “GRE IP Hdr” in tunnel mode(in pink), which includes the source/dest address of the GRE tunnel.
In both modes, the “New IP Hdr” is the same as “GRE IP Hdr”, which includes the source/dest address of the tunnel.
This is an important point to keep in mind, because on AWS and other cloud platforms, the instance itself may not be aware of the public IP address, there is NAT performed by the cloud platform to map a private IP to a public IP and vice versa.
Below is an example set up of Cisco CSR1000V on AWS, AWS performs NAT for the outgoing interface (GigabitEthernet0) to translate from private IP 10.10.1.100 to 13.236.166.176.
Note that in tunnel mode, the “GRE IP Hdr” and original IP package are encrypted, and AWS NAT device will not translate for “GRE IP Hdr”, only the source IP of “New IP Hdr” is translated to the external IP.
So the source IP inside the “GRE IP Hdr” is still the private IP 10.10.1.100, which is the only IP that the Cisco instance is aware of.
This is the cause of problem, as the peer gateway sets up GRE Tunnel with destination address 13.236.166.176, and it expects to see the source address in “GRE IP Hdr” as 13.236.166.176 when it receives the packet, but instead it sees 10.10.1.100, there is a mismatch.
Therefore, to make GRE over IPsec work on AWS, we need to use IPsec transport mode.
Debug
Below is the debug log captured on the peer Cisco device when IPsec mode is the default “mode tunnel”. The error is “IPSec policy invalidated proposal with error 32“.
The local_proxy/remote_proxy defines the interesting traffic to be encrypted. In GRE over IP, the traffic to be encrypted is between the source/dest address of the tunnel. There is no need to define an access list, as the GRE tunnel source/dest addresses are automatically considered to be the proxy ACL.
Jan 23 02:32:25.719: IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 31.xxx.xxx.48:0, remote= 13.236.166.176:0,
local_proxy= 31.xxx.xxx.48/255.255.255.255/47/0,
remote_proxy= 10.10.1.100/255.255.255.255/47/0,
protocol= ESP, transform= esp-aes 256 esp-sha-hmac (Tunnel-UDP),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 256, flags= 0x0
Jan 23 02:32:25.719: map_db_find_best did not find matching map
Jan 23 02:32:25.719: map_db_find_best did not find matching map
Jan 23 02:32:25.719: ISAKMP-ERROR: (1036):IPSec policy invalidated proposal with error 32
Jan 23 02:32:25.719: ISAKMP-ERROR: (1036):phase 2 SA policy not acceptable! (local 31.xxx.xxx.48 remote 13.236.166.176)
Jan 23 02:32:25.719: ISAKMP: (1036):set new node -638729155 to QM_IDLE
Jan 23 02:32:25.719: ISAKMP: (1036):Sending NOTIFY PROPOSAL_NOT_CHOSEN protocol 3
#011spi 288474424, message ID = 3656238141
Jan 23 02:32:25.719: ISAKMP-PAK: (1036):sending packet to 13.236.166.176 my_port 4500 peer_port 4500 (R) QM_IDLE
Jan 23 02:32:25.719: ISAKMP: (1036):Sending an IKE IPv4 Packet.
Jan 23 02:32:25.719: ISAKMP: (1036):purging node -638729155
Jan 23 02:32:25.719: ISAKMP-ERROR: (1036):deleting node -1097053243 error TRUE reason "QM rejected"
After changing the mode to “mode transport”, we can see now the IPs are matching, and ISAKMP and IPsec are established.
Jan 23 02:35:27.229: IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 31.xxx.xxx.48:0, remote= 13.236.166.176:0,
local_proxy= 31.xxx.xxx.48/255.255.255.255/47/0,
remote_proxy= 13.236.166.176/255.255.255.255/47/0,
protocol= ESP, transform= esp-aes 256 esp-sha-hmac (Transport-UDP),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 256, flags= 0x0
Jan 23 02:35:27.229: Crypto mapdb : proxy_match
#011src addr : 31.xxx.xxx.48
#011dst addr : 13.236.166.176
#011protocol : 47
#011src port : 0
#011dst port : 0
Jan 23 02:35:27.229: (ipsec_process_proposal)Map Accepted: Tunnel0-head-0, 65537
Jan 23 02:35:27.229: ISAKMP: (1038):processing NONCE payload. message ID = 1516400312
Jan 23 02:35:27.229: ISAKMP: (1038):processing ID payload. message ID = 1516400312
Jan 23 02:35:27.229: ISAKMP: (1038):processing ID payload. message ID = 1516400312
Jan 23 02:35:27.229: ISAKMP: (1038):received payload type 21
Jan 23 02:35:27.229: ISAKMP: (1038):received payload type 21
Jan 23 02:35:27.229: ISAKMP: (1038):Node 1516400312, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
Jan 23 02:35:27.229: ISAKMP: (1038):Old State = IKE_QM_I_QM1 New State = IKE_QM_IPSEC_INSTALL_AWAIT
Jan 23 02:35:27.229: IPSEC(key_engine): got a queue event with 1 KMI message(s)
Jan 23 02:35:27.229: Crypto mapdb : proxy_match
#011src addr : 31.xxx.xxx.48
#011dst addr : 13.236.166.176
#011protocol : 47
#011src port : 0
#011dst port : 0
Jan 23 02:35:27.229: IPSEC(crypto_ipsec_create_ipsec_sas): Map found Tunnel0-head-0, 65537
Jan 23 02:35:27.229: IPSEC(get_old_outbound_sa_for_peer): No outbound SA found for peer 113210DC
Jan 23 02:35:27.229: IPSEC(create_sa): sa created,
(sa) sa_dest= 31.xxx.xxx.48, sa_proto= 50,
sa_spi= 0x1A49EC7A(441052282),
sa_trans= esp-aes 256 esp-sha-hmac , sa_conn_id= 2005
sa_lifetime(k/sec)= (4608000/3600),
(identity) local= 31.xxx.xxx.48:0, remote= 13.236.166.176:0,
local_proxy= 31.xxx.xxx.48/255.255.255.255/47/0,
remote_proxy= 13.236.166.176/255.255.255.255/47/0
Jan 23 02:35:27.229: IPSEC(create_sa): sa created
(sa) sa_dest= 13.236.166.176, sa_proto= 50,
sa_spi= 0x5512E8E2(1427302626),
sa_trans= esp-aes 256 esp-sha-hmac , sa_conn_id= 2006
sa_lifetime(k/sec)= (4608000/3600),
(identity) local= 31.xxx.xxx.48:0, remote= 13.236.166.176:0,
local_proxy= 31.xxx.xxx.48/255.255.255.255/47/0,
remote_proxy= 13.236.166.176/255.255.255.255/47/0
Jan 23 02:35:27.229: IPSEC: Expand action denied, notify RP
Jan 23 02:35:27.229: ISAKMP-ERROR: (0):Failed to find peer index node to update peer_info_list
Jan 23 02:35:27.229: ISAKMP: (1038):Received IPSec Install callback... proceeding with the negotiation
Jan 23 02:35:27.229: ISAKMP: (1038):Successfully installed IPSEC SA (SPI:0x1A49EC7A) on Tunnel0
Jan 23 02:35:27.229: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up