XSJ Boardfarm - Ethernet Testing Setup

📚 Introduction

This page shows various setups for board-to-board Ethernet testing.

🔗 Board-to-Board setup

Show entries

🔍 Search boards:

Board	Connection Type	Port	CR
VEK385-7 & VEK385-8	QSFP	`QSFP1`	🔗 CR-1253164 WTS-WWG-Ethernet-Testing - Need a VEK385 to VEK385 connection via QSFP CLOSED
VCK190-13 & VCK190-14	QSFP	`QSFP0`	🔗 CR-1198548 WTS-WWG-Ethernet-Testing - Need a VCK190 to VCK190 connection via QSFP CLOSED
VPK120-5 & VPK120-6	QSFP	`QSFP2`	🔗 CR-1204096 WTS-WWG-Ethernet-Testing - VPK120 to VPK120 QSFP connection CLOSED
VPK180-5 & VPK180-6	QSFP	`QSFP4`	🔗 CR-1226912 WTS-WWG-Ethernet-Testing - VPK180 to VPK180 connection via QSFP CLOSED
VCK190-13 & VCK190-14	SFP	`SFP0`	🔗 CR-1184041 WTS-WWG-Ethernet-Testing - VCK190 board to board setup CLOSED
VCK190-9 & VCK190-10 (VMK180)	SFP	`SFP0`	🔗 CR-1212175 WTS-WWG-Ethernet-Testing - Need a VMK180 to VMK180 connection via SFP CLOSED
VEK280-10 & VEK280-11	SFP	`SFP0`	🔗 CR-1232780 WTS-WWG-Ethernet-Testing - VEK280 board to board setup CLOSED
ZCU102-5 & ZCU102-6	SFP	`SFP0`	🔗 CR-1202007 WTS-WWG-Ethernet-Testing - Need ZCU102 SFP connection for 10G ethernet testing CLOSED
ZCU111-3 & ZCU111-4	SFP	`SFP2`	🔗 CR-1242683 WTS-WWG-Ethernet-Testing - Need ZCU111 SFP connection for 25G ethernet testing CLOSED
XHD-ZCU670-5 & XHD-ZCU670-6 📌 See XSJ Boardfarm - XHD boards in XSJ Machine to use this setup	SFP	`SFP0`	🔗 CR-1241140 WTS-WWG-Ethernet-Testing - Need ZCU670 SFP connection for 1G/10G ethernet testing CLOSED

🖥️ Board-to-NIC

⚠️ Important: Only three of these boards can be used at a time with Mellanox. Please comment in the CR to change the connection.

Show entries

🔍 Search connections:

#	Board	NIC	Host	CR
1	VCK190-25	Mellanox Connect X5	`xsjzebu13 - eth4`	🔗 CR-1222776 WTS-WWG-Ethernet-Testing - VCK190 connection with a Mellanox NIC CLOSED
2	VMK180-?	Mellanox Connect X5	`-`	🔗 CR-1232357 WTS-WWG-Ethernet-Testing - VCK180 connection with a Mellanox NIC CLOSED
3	VPK180-?	Mellanox Connect X5	`-`	🔗 CR-1232343 WTS-WWG-Ethernet-Testing - VPK180 connection with a Mellanox NIC CLOSED
4	VPK120-11	Mellanox Connect X5	`xsjzebu13 - eth5`	🔗 CR-1231428 WTS-WWG-Ethernet-Testing - VPK120 connection with a Mellanox NIC CLOSED
5	VCU118-5	Mellanox Connect X5	`xsjzebu13 - eth7`	🔗 CR-1237791 WTS-WWG-Ethernet-Testing - VCU118 connection with a Mellanox NIC CLOSED

🔌 Connection Diagram - xsjzebu13 Host

graph TB subgraph Host["🖥️ xsjzebu13 Host Server"] direction LR subgraph Mel0["🔶 Mellanox0"] eth4["eth4"] eth5["eth5"] end subgraph Mel1["🔶 Mellanox1"] eth6["eth6"] eth7["eth7"] end end VCK190["📟 VCK190-25"] --> eth4 VMK120["📟 VMK120-11"] --> eth5 OPEN["⚠️ OPEN"] -.-> eth6 VCU118["📟 VCU118-5"] --> eth7 style Host fill:#1e3a8a,stroke:#1e40af,stroke-width:3px,color:#fff style Mel0 fill:#f97316,stroke:#ea580c,stroke-width:2px,color:#fff style Mel1 fill:#f97316,stroke:#ea580c,stroke-width:2px,color:#fff style eth4 fill:#fed7aa,stroke:#fdba74,stroke-width:2px,color:#000 style eth5 fill:#fed7aa,stroke:#fdba74,stroke-width:2px,color:#000 style eth6 fill:#fed7aa,stroke:#fdba74,stroke-width:2px,color:#000 style eth7 fill:#fed7aa,stroke:#fdba74,stroke-width:2px,color:#000 style VCK190 fill:#0369a1,stroke:#0284c7,stroke-width:2px,color:#fff style VMK120 fill:#0369a1,stroke:#0284c7,stroke-width:2px,color:#fff style OPEN fill:#64748b,stroke:#94a3b8,stroke-width:2px,stroke-dasharray: 5 5,color:#fff style VCU118 fill:#0369a1,stroke:#0284c7,stroke-width:2px,color:#fff

⚙️ Setup details

🔌 Cable information

Model	Type
SFP28-25G	SFP
2336313-2	QSFP

🔶 Mellanox Setup

Currently, we have 2 Mellanox in this setup as shown in the diagram above.

1  ajayad@xsjzebu13 ~ - $ lspci | grep Mella
2  25:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
3  25:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
4  81:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
5  81:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]

Below are the interfaces. Each card has 2 ports:

1   ajayad@xsjzebu13 ~ - $ ifconfig
2   eth6: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
3           ether 6c:b3:11:88:0f:ea  txqueuelen 1000  (Ethernet)
4           RX packets 0  bytes 0 (0.0 B)
5           RX errors 0  dropped 0  overruns 0  frame 0
6           TX packets 0  bytes 0 (0.0 B)
7           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

8   eth5: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
9           ether 6c:b3:11:88:0f:eb  txqueuelen 1000  (Ethernet)
10          RX packets 0  bytes 0 (0.0 B)
11          RX errors 0  dropped 0  overruns 0  frame 0
12          TX packets 0  bytes 0 (0.0 B)
13          TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

14  eth6: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
15          ether 6c:b3:11:88:10:06  txqueuelen 1000  (Ethernet)
16          RX packets 0  bytes 0 (0.0 B)
17          RX errors 0  dropped 0  overruns 0  frame 0
18          TX packets 0  bytes 0 (0.0 B)
19          TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

20  eth7: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
21          ether 6c:b3:11:88:10:07  txqueuelen 1000  (Ethernet)
22          RX packets 0  bytes 0 (0.0 B)
23          RX errors 0  dropped 0  overruns 0  carrier 0  collisions 0

⏱️ How to program the GT ref clock

📥 Download PDF

Slide 1 of 12

💡 Tip: Use arrow keys (← →) to navigate slides

🔧 Workaround to program the GT REF clock in VCK190

There have been challenges in programming the GT REF clock on boardfarm boards, particularly since the BEAM tool is not available. Additionally, using the system-controller image app (sc_app) has proven unreliable.

The clock can be programmed using the PetaLinux IDT driver as a workaround.

⚠️ Important: Before downloading the PetaLinux image via JTAG, ensure the board is disconnected from the Vivado Hardware Manager.

📊 322.2656MHz

Use the provided image (built using the VCK190 BSP to program 322.x MHz) and download it to the board as shown below. The image is located at /public/cases/gt_refclk_program/xilinx-vck190-2024.2

Aquire the board using systest, start hw_server, and connect com0. For example, below is vck190-15:

Terminal 1:

1  [vck190-15] Systest# loadmodule "2024.2_released"
2  [vck190-15] Systest# hw_server
3  Vivado hardware server is running on: chanterelle12:3121
4
5  [vck190-15] Systest# connect com0
6  Connecting to device com0. Use Ctrl-\ to escape.

Terminal 2:

On a separate terminal, download the petalinux to the board:

1  ts -petalinux 2024.2
2  cd /public/cases/gt_refclk_program/xilinx-vck190-2024.2
3  petalinux-boot --jtag --kernel --rootfs ./images/linux/rootfs.cpio.gz.u-boot --hw_server tcp:chanterelle12:3121

Verify in PetaLinux

In Terminal 1, once the image has been fully downloaded, the PetaLinux login screen will appear. It is possible to verify whether the clock is programmed successfully. Login by using username petalinux and password petalinux. Type dmesg | grep idtcm. You should see the following [3.864659] 8a3400x-phc 8a3400x-phc.2.auto: requesting firmware 'idtcm.bin'. At this point, press Ctrl-\ to exit back to systest, then run reset. Your board will now be ready with the 322.x MHz clock.

********************************************************************************
The PetaLinux source code and images provided/generated are for demonstration pu
Please refer to https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2741928025
for more details.
********************************************************************************
PetaLinux 2024.2+release-S11061705 PetaLinux ttyAMA0

PetaLinux login: petalinux
********************************************************************************
The PetaLinux source code and images provided/generated are for demonstration pu
Please refer to https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2741928025
for more details.
********************************************************************************
PetaLinux 2024.2+release-S11061705 PetaLinux ttyAMA0

xilinx-vck190-20242:~$ petalinux
You are required to change your password immediately (administrator enforced).
New password:
Retype new password:
[   84.545685] audit: type=1006 audit(2150864578.264:2): pid=751 uid=0 old-auid=
[   84.558284] audit: type=1300 audit(2150864578.264:2): arch=c00000b7 syscall=
[   84.584996] audit: type=1327 audit(2150864578.264:2): proctitle="(systemd)"

xilinx-vck190-20242:~$
xilinx-vck190-20242:~$ sudo su

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

For security reasons, the password you type will not be visible.

Password:
xilinx-vck190-20242:/home/petalinux# dmesg | grep idt
[   3.864659] 8a3400x-phc 8a3400x-phc.2.auto: requesting firmware 'idtcm.bin'
######THIS is where I pressed CTRL-\######
PetaLinux:/home/petalinux# [vck190-15] Systest#
[vck190-15] Systest# reset
[vck190-15] Systest# connect com0

Now it is ok to connect the board in the HW Manager and download the PDI and ELF file.

The TCS file was obtained from here: 📄 AMD Customer Community

📊 156.25Mhz

For any other clock frequency, such as 156.25MHz, the provided PetaLinux project can be modified.

Copy the Petalinux project to your workspace: cp -rf /public/cases/gt_refclk_program/xilinx-vck190-2024.2 <YOUR WORKSPACE>
Generate a new tcs file using 📝 Timing Commander. Skip this if you already have tcs file; for example, the 156.25MHz file is already included in system-controller. It's possible to download the file using scp to your local workspace.
Use tcs2bin to generate idtcm.bin

1  chmod +x tcs2bin
2  ./tcs2bin -i <file_name.tcs> -o idtcm.bin

📥

Downloadable Resource

Click to download the tcs2bin tool

Download

Replace the file inside xilinx-vck190-2024.2/project-spec/meta-user/recipes-apps/idtcm/files/
Run the following commands:

1  petalinux-build -x mrproper && petalinux-build
2  petalinux-package --boot --plm --psmfw --u-boot --dtb --force
3  petalinux-boot --jtag --kernel --rootfs ./images/linux/rootfs.cpio.gz.u-boot --hw_server tcp:chanterelle12:3121

Now, your board will be programmed with the clock file generated above. Verify as shown in section 📂 XSJ Boardfarm - WTS-WWG - Ethernet Testing Setup | Verify in PetaLinux

🖥️ XHD boards in XSJ Machine

1   ssh xhdbf1
2   /proj/systest/bin/systest xhd-zcu670-5
3   Job <9467010> is submitted to queue <hwboard>.
4   <<Waiting for dispatch ...>>
5   <<Starting on xhdbfarmkaa6>>
6   -- Systest server on host xhdbfarmkaa6 started on May 28, 2025 8:19:04 AM ------
7   This setup has the following serial ports:
8       com0, com1, com2
9   Board Farm Confluence page: https://amd.atlassian.net/wiki/spaces/XPS/pages/603288468
10  You can see the list of commands by typing "help"
11  -----------------------------------------------------------------------------
12  [xhd-zcu670-5] Systest#