Connecting two gigabit-capable devices via 4-wire UTP cable is an abomination, but sometimes we have to live with it (e.g. when a greedy ISP decides to save a few cents and pulls a cheap cable to your apartment).
The fun starts when the devices try to negotiate Ethernet connection speed. Autonegotiation pulses are essentially 10BASE-T and use only 4 wires with no checking of cable category, even CAT3 is enough. So autonegotiation will "succeed" and will settle on 1Gbit since it's supported by both ends, even though there is not enough physical conductors for it. The link will not work.
To negotiate a working connection instead we need to remove 1Gbit from advertised link modes at least on one end. In Linux it should be rather straightforward with ethtool:
$ ethtool -s $IFACE advertise 0x00f
0x00f is a sum of hex values for all 10Mbit and 100Mbit modes
(0x001 + 0x002 + 0x004 + 0x008), explicitly excluding 1Gbit and everything
above that. Tweaking advertised link modes is better than forcing a particular
connection speed (ethtool -s $IFACE speed 100) because in absence of
autonegotiation advertisement the other side may play it safe and switch
to half-duplex.
That's the theory. In practise however, hardware is difficult. Drivers for NICs are tricky and sometimes incomplete. On D-Link DIR-825 changes made by ethtool command did not stick: after a short hiccup NIC would immediately return back to default settings, ignoring ethtool input completely.
But all is not lost. Through trial and error I found a sequence that worked. Don't ask me why, I'm not cool enough yet to understand kernel drivers code. Here is what worked for me:
# Excerpt from /etc/rc.local
# (order of commands and delays matter!)
WAN=eth1
ethtool -s $WAN autoneg off
sleep 1
ethtool -s $WAN speed 100
sleep 1
ethtool -s $WAN advertise 0x00f
Which results in a proper NIC configuration:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 4
Transceiver: external
Auto-negotiation: on
Current message level: 0x000000ff (255)
drv probe link timer ifdown ifup rx_err tx_err
Link detected: yes
Let's hope some variation of this will work for your misbehaving device too! Good luck!