← AI Foundry lab dashboard › Worklog
Covers Wed 7 Oct 2026 (week 41), 15:09–15:31 PDT: aifoundry1, aifoundry2 and aifoundry3 at Studio 45. Written the same afternoon from timed transfers between every pair of machines, run from aifoundry1.
Update, 14 October. Step 1 of To make it usable is done: aifoundry1 and aifoundry2 got fixed private addresses on their wired ports on 11 October and aifoundry3 on 13 October, and each machine then reached the other two over the cable in about 0.5 ms. The link is not yet dependable: all three ports lost it at about 22:35 on 13 October and got it back at 13:56 on 14 October, within seven seconds of each other, which points at the switch or its power, and in between the machines talked over WiFi. Each port has lost its link twice since its machine last started, so the cable and its switch still need a look on site. The measurements below are from 7 October and stand.
The three machines are cabled to one wired network, and it works. Every port came up at 1 Gb/s, every machine reaches the other two, and one TCP stream moves 929 Mb/s (116 MB/s) in every direction: the most a gigabit link carries. WiFi between the same machines moves a median 31 Mb/s (19–45), so the cable is about 30 times faster. A round trip takes 0.26 ms on the cable and about 5.9 ms over WiFi.
But nothing uses the cable yet. The wired ports have no IP address. Nothing on that network hands addresses out (no DHCP server), so each machine's NetworkManager asks for 45 seconds, four times, gives up for 5 minutes and starts again. Until each port gets an address, all traffic between the machines, ssh and Tailscale included, still goes over WiFi. These tests used the ports' automatic IPv6 link-local addresses, which exist only while NetworkManager is asking: about 3 minutes in every 8. The fix is a fixed address on each wired port, a one-time step for whoever has root on the machines.
Every pair, every path
Speed in Mb/s, one TCP stream for 8 seconds, counted by the receiver. Tailscale between these machines connects directly over the WiFi network, so it measures WiFi plus encryption.
| From → to | Ethernet | WiFi | Tailscale |
|---|---|---|---|
| aifoundry1 → aifoundry2 | 928.5 | 18.8 | 22.2 |
| aifoundry2 → aifoundry1 | 928.5 | 22.6 | 28.0 |
| aifoundry2 → aifoundry3 | 928.5 | 44.8 | 39.0 |
| aifoundry3 → aifoundry2 | 928.5* | 32.9 | 31.9 |
| aifoundry3 → aifoundry1 | 928.5 | 29.1 | 26.6 |
| aifoundry1 → aifoundry3 | 928.4* | 37.6 | 35.1 |
* The first run in this direction overlapped a NetworkManager restart; the best of its runs is shown (see How it was measured).
Round trip in ms, median (99th percentile in brackets):
| Pair | Ethernet | WiFi | Tailscale |
|---|---|---|---|
| aifoundry1 ↔ aifoundry2 | 0.26 (0.40) | 5.3 (48.8) | 5.2 (18.1) |
| aifoundry2 ↔ aifoundry3 | 0.25 (0.30) | 5.9 (47.6) | 5.9 (31.6) |
| aifoundry3 ↔ aifoundry1 | 0.26 (0.44) | 6.4 (25.4) | 5.0 (17.3) |
- Four streams at once move the same 930 Mb/s as one, on every pair: the link is the limit, not the machines or the software.
- Both directions at once (aifoundry1 → 2 and 2 → 1 together): 925 and 925 Mb/s. The link is full duplex: each direction keeps the whole gigabit.
- All three links at once (1 → 2, 2 → 3 and 3 → 1 together): 925, 925, 925 and 925, 925, 925 Mb/s in the 2 runs that no NetworkManager restart touched, about 2,775 Mb/s in all. The switch gives every port its full speed at the same time. In the 4 runs a restart touched, two of the three links dropped to 558–755 Mb/s.
What it means
Time to copy a file from one machine to another, at the speeds measured:
| File | Ethernet (929 Mb/s) | WiFi (31 Mb/s) |
|---|---|---|
| 1 GB (a model checkpoint) | 9 s | 4 min |
| 10 GB | 86 s | 43 min |
| 100 GB | 14 min | 7.2 h |
WiFi is also uneven: its slowest round trips took up to 48.8 ms, against 0.44 ms on the cable.
Why 1 Gb/s
Each machine's wired port is an Aquantia AQC107, a 10 Gb/s controller that also runs at 2.5 and 5 Gb/s, and each offers all of those speeds. All three links settled at 1 Gb/s, so the switch (or the cabling) is gigabit. A 10 Gb/s switch would raise the ceiling up to ten times.
To make it usable
- A fixed address on each wired port (root, one command per machine) and a name for each, such as aifoundry2-eth. NetworkManager then stops asking for an address every few minutes.
- Tailscale should then pick the cable by itself. It offers every address a machine has and uses
the fastest path it finds between two machines, so
ssh aifoundryN, which goes through Tailscale, would ride the cable. To be checked once the ports have addresses. - Plain ssh to a wired address needs a key. Between the lab machines, ssh works today only through
Tailscale; to the wired address it was refused for lack of a key in
~/.ssh/authorized_keys.
How it was measured
- The cabling. Each port showed a carrier at 1 Gb/s full duplex. Pings between every pair over the link-local addresses were answered in about 0.5 ms. No port counted an error or a dropped packet; the frames are the standard 1,500 bytes.
- The tool.
netbench.py, a small Python TCP sender and receiver: each test runs 8 s after a 1 s warm-up that is discarded, and the receiver counts the bytes. The round trip is a 64-byte message and its echo on one TCP connection, repeated 2,000 times on Ethernet and 200 times over WiFi. - When. WiFi and Tailscale at 15:09–15:12 PDT; Ethernet at 15:13–15:31, each test waiting until both ends had their address.
- Slow runs. 7 of the 20 Ethernet runs had a link below 900 Mb/s (558–761), and every one overlapped a NetworkManager restart, by the machines' logs or by watching the ports' addresses during the run. All 12 runs that no restart touched ran at full speed, and so did 1 that a restart touched. A restart re-sets the port's address, but it is not only that port that slows: in the all-links runs a restart on one machine sometimes slowed the link between the other two. It does not bounce the cable (the ports' carrier counters did not move). Fixed addresses end the restarts. The table shows each direction's best run. * marks a direction whose first run was one of these.
- WiFi moves with everyone else's traffic and with the access point a machine is on. These are a few minutes' sample, not a long-term average.
The tool, the scripts and every result are in
docs/reports/data/2026-10-07-lab-ethernet
of the lab repository, with this page as docs/reports/2026-10-07-lab-ethernet.html.