[Planetlab-users] sendto: operation not permitted

[Neil Spring]

Patrick,

Explaining this one is going to take more understanding of the  
connection tracking underneath than I have.  I can see it as sketchy  
to send the ack of the handshake before seeing the sequence number  
(in the syn ack) to acknowledge (the acks you send aren't kosher), so  
I could believe that the code would prevent it.  Holding the "syn- 
sent" connection state for two minutes (an MSL) seems a reasonable  
explanation of the last excerpt. (that is, after the connection state  
goes away, vnet lets you send an ack that isn't part of a connection- 
in-setup.

On the other hand, denying the acks doesn't seem to be protecting  
against sending a bad ack (first excerpt), so perhaps if the EPERM  
can be suppressed, vnet's behavior might be more predictable.   At  
least it's deterministic! :)

I think this one requires Mark.

-neil

On Nov 12, 2006, at 9:40 PM, Patrick Verkaik wrote:
> The program binds a socket to a local port and (through a separate  
> raw socket) sends a SYN and then immediately an ACK packet to  
> REMOTE_ADDR (132.239.17.226) port REMOTE_TCP_PORT (33445). For both  
> packets if it encounters EPERM it sleeps a second and retries (and  
> keeps doing so until EPERM disappears).
>
> This is the tcpdump output when the remote host is allowed to send  
> back a SYN/ACK:
> 20:53:18.068949 IP 132.239.17.224.50227 > 132.239.17.226.33445: S  
> 10:10(0) win 20502
> 20:53:18.079683 IP 132.239.17.226.33445 > 132.239.17.224.50227: S  
> 905277696:905277696(0) ack 11 win 5712 <mss 1428>
> 20:53:19.071081 IP 132.239.17.224.50227 > 132.239.17.226.33445: .  
> ack 3389689600 win 20502
>
> (The program gets one EPERM since its first attempt at sending ACK  
> precedes the SYN/ACK from the remote host.)
>
> Tcpdump output when the remote host sends back RST instead of SYN/ACK:
> 21:05:26.991199 IP 132.239.17.224.52034 > 132.239.17.226.33445: S  
> 10:10(0) win 20502
> 21:05:26.991347 IP 132.239.17.226.33445 > 132.239.17.224.52034: R  
> 0:0(0) ack 11 win 0
> 21:05:26.991979 IP 132.239.17.224.52034 > 132.239.17.226.33445: .  
> ack 0 win 20502
>
> So far so good. However, if I suppress the SYN/ACK from the remote  
> host I get this:
>
> 21:09:03.005807 IP 132.239.17.224.52554 > 132.239.17.226.33445: S  
> 10:10(0) win 20502
> 21:11:03.555572 IP 132.239.17.224.52554 > 132.239.17.226.33445: .  
> ack 0 win 20502
>
> For a full two minutes the program repeatedly gets EPERM as it  
> tries to send the ACK. After the two minutes have passed the ACK  
> finally goes through.
>
> In the above the program is running on planetlab1.ucsd.edu. When  
> run on planet-lab2.cs.ucr.edu it shows exactly the same behaviour.
>
> Can you confirm that we're doing everything the way we should?
>
> 	Thanks,
> 	Patrick
>
> On Thu, 9 Nov 2006, Neil Spring wrote:
>
>> Patrick,
>>
>> PlanetLab's vnet works on the assumption that you can send tcp  
>> packets so long as the source port is one your slice "owns" via  
>> opening a socket and calling bind, and that you can only receive  
>> tcp packets to a destination port that you "own" in the same way.
>>
>> We send gobs of tcp packets, without EPERM on sendto.   If it's  
>> letting a packet through after 100,000 retries, that sounds like a  
>> bug.
>>
>> Planetlab rate limiting has been found to delay packets by 30  
>> seconds (I'm as surprised as anyone, but I heard this from a  
>> reputable source on sunday). I'd guess that if you were really  
>> blowing the queue, you'd get ENOBUFF rather than EPERM.
>>
>> I'm pretty sure Mark wrote a vnet faq, or at least the  
>> documentation should get you pointed in the right direction.
>>
>> -neil
>>
>> On Nov 9, 2006, at 2:58 AM, Patrick Verkaik wrote:
>>
>>> Hi,
>>> I have a question about the following sendto() behaviour that I  
>>> don't quite understand.
>>> We're sending raw TCP packets using sendto() but getting  
>>> intermittent EPERM errors. We've found that repeatedly retrying  
>>> the sendto() (with exactly the same packet) until it no longer  
>>> gives EPERM eventually gets the packet through. During a short  
>>> connection (perhaps lasting 10 seconds and sending across less  
>>> than 100 bytes of data) we sometimes see about 100,000 failed  
>>> sendto() attempts (with a usleep(1) separating the attempts  
>>> roughly 20,000 attempts).
>>> Another curious fact is that we only see this behaviour when  
>>> we're tunneling the reverse TCP traffic into the sending host.  
>>> The host therefore doesn't see e.g. SYN/ACK packets coming back  
>>> in response to outgoing SYNs.
>>> Can anyone explain this? Is this how Planetlab implements rate  
>>> limiting or prevents SYN-flooding attacks being launched from  
>>> Planetlab?
>>> (Btw: we're running as root and using the node's IP address as  
>>> source IP address.)
>>>
>>> 	Thanks,
>>>
>>> 	Patrick Verkaik
>>> 	Barath Raghavan
>>> _______________________________________________
>>> Users mailing list: Users at lists.planet-lab.org
>>> https://lists.planet-lab.org/mailman/listinfo/users
>>
>> _______________________________________________
>> Users mailing list: Users at lists.planet-lab.org
>> https://lists.planet-lab.org/mailman/listinfo/users
>
> -- 
>
> 	Patrick
> <example.c>