MAPMTFEB
MAPMTFEB is a readout board designed to receive binary hit data from 128 FE channels. The digital data is formatted in a Spartan6 FPGA and send to an Ethernet readout network in LHCb multi-event packet format. The readout is controlled via external signals connected via a 2x5 0.1 inch header. The board geometry is optimised for close-packing of Hamamatsu R11265 MaPMTs with each MAPMTFEB connected to two MaPMTs. However, the firmware can be adapted to support other applications if desired.
The readout architecture is very similar to the LHCb readout but adapted to facilitate the merging of TimePix tracking telescope data. Events are tagged with a timestamp (BXID) and event ID and the data are sent out as multi-event packets (MEPs). MEP building is done using internally generated readout type (ROT) and MEP destination commands. Readout is controlled with external triggers (TRG). In addition there is an external start-of-frame (SOF) signal which acts rather like the bunch count reset signal. It resets the timestamp counter. An SOF counter is also sent with the data.
A single 5V/1A power supply connected to the red (+5V) and black (0V) 2mm sockets is required.
The MAPMTFEB data and configuration interfaces use the same 100baseTX connection. Standard network hardware can be used to connect to a PC. However, the MAPMTFEB data packets must not be routed through a general-purpose network. Use either a PC with a separate NIC for the data and configuration or, if several boards must be used simultaneously, connect them via a dedicated network switch. A managed switch may be needed because it may be necessary to disable some features such as "storm control" or to create a VLAN.
TRG and SOF signals (3V3 LVTTL) are received on the 2x5 pin header. Note that there is no load termination at the FPGA. TTL outputs of NIM-TTL converters are designed to drive 50 Ohm cables so it will be necessary to add terminators at the MAPMTFEB end of the cable if NIM-TTL converters are used to source these signals.
These and other signals at the 2x5 pin header are summarised in the table. Z indicates an undefined, high-impedance pin.
1 |
SCL |
2 |
SDA |
3 |
AUX1P |
4 |
AUX1N |
5 |
TRG |
6 |
Z |
7 |
SOF |
8 |
Z |
9 |
+5V |
10 |
GND |
Status information may be accessed via the SMB SCL/SDA pins.
Front-end data are received through a 6x30 SAMTEC SEAF connector. This connector also supplies LV power to the front-end. Some pins are connected to FPGA general purpose IO pins and can be used for front-end configuration and monitoring.
Configuration and readout are currently possible only under Linux.
The MEP events sent via Ethernet are received by the program feb-mdfwriter which reads from a raw socket (and therefore requires elevated permissions). The program extracts the MEPs, reformats the data and writes in MDF format. The MDF filename is tagged with a run number given as a command line argument. The program will exit if a file with the same name already exists to avoid accidentally overwriting data. The filename is constructed by substituting the run number into a sprintf-style format specification defined using the environment variable FEB_FILENAME_MODEL. For example:
FEB_FILENAME_MODEL="/var/data/mapmtfeb-%06d.mdf"
Recording can then be started with the command:
sudo -E feb-mdfwriter --run <run>
Configuration is done using the program feb-cfg which builds a configuration packet and sends it to MAPMTFEB over its Ethernet interface. Some environment variables must be set that describe the local network (the values below are examples only):
| FEB_MAC_DESTINATION | 00:01:02:03:04:05 |
| FEB_MAC_SOURCE | 02:F1:F2:F3:F4:F5 |
| FEB_IP_DESTINATION | 192.168.244.1 |
| FEB_IP_SOURCE | 192.168.244.2 |
IP addresses on a private network should be used.
Other parameters are hardcoded in the program while some can be set via command line arguments. For example:
| sudo -E feb-cfg | --id <idx> |
| --latency <lat> | |
| --trigger-delay <del> | |
| --strobe-length <len> | |
| --invert-aux <mask> | |
| --l1id <id> | |
| --mep-max-fragment <size> | |
| --mep-hwm <hwm> | |
| --start |
Each MAPMTFEB has a unique hardware ID. The --id parameter is an index to a hardcoded table containing the hardware ID which is sent with the configuration data.
By default, feb-cfg puts MAPMTFEB into a state where it does not respond to triggers and does not send data. Use the --start option to enable the processing of triggers and transmission of data to the network.
Most of the steps described here will require elevated privileges.
feb-cfg and feb-mdfwriter may be added to the /etc/sudoers configuration so that they can be executed by unprivileged users.
The current MAPMTFEB firmware supports only 100baseTX Ethernet. The immediate upstream network port should be set to autonegotiate but to advertise only 100Mbit full-duplex. This can usually be done through the management interface of a switch or using the ethtool program for a NIC in a Linux PC e.g.:
ethtool -s eth1 advertise 0x008
To make this setting persistent across reboots the command could be added in a suitable udev rules file in /etc/udev/rules.d/.
MAPMTFEB does not implement a compliant MAC controller so it may also be necessary to add a manual entry in the ARP cache on the readout PC.
arp -s $FEB_IP_SOURCE $FEB_MAC_SOURCE
This setting is not persistent across reboots.
You may also need to change your firewall configuration to allow the MAPMTFEB packets through on the DAQ interface. For example, the following command inserts an ACCEPT rule at the beginning of the INPUT chain for interface eth1 that will accept packets coming from any MAPMTFEB on the network 192.168.244.0 and having protocol 242:
| iptables -I INPUT | -i eth1 -p 242 |
| -s 192.168.244.0/24 | |
| -j ACCEPT |
This will not be persistent across reboots in all Linux environments.
The different time of arrival of the detector signals and the corresponding trigger can be accommodated by the trigger_delay and latency configuration parameters. The normal case would be that the detector signals arrive at MAPMTFEB before the trigger. Then the latency parameter can be used to delay the detector signals to bring them into alignment with the trigger. On the other hand, in some set-ups, it may happen that the detector signals arrive later than the trigger. In this case, use the trigger_delay parameter to delay the trigger.
Important note: To avoid data corruption when reading the latency pipeline it is important that latency and trigger_delay are not both set to values close to zero.
The correct timing is usually found by taking a series of data points at different latency (or trigger_delay) settings while monitoring the number of hits seen in the detector. The correct timing is achieved when the number of hits is maximum. Remember that it is the difference between trigger_delay and latency that is important so only one of these parameters should need to be scanned.
It may help when finding the correct timing to use a longer than usual strobe_length value. Once the approximate timing is found, strobe_length can be reduced and a mini-scan done to home in. Note however that a longer strobe_length will also integrate more noise so may not help in a noisy environment.
Status is currently only accessible through the SMB interface. Offset is in bytes.
Offset |
Content |
7:0 |
dna |
8 |
"000000"ðclkdcm_locked&gclkdcm_locked |
9 |
"0000000"&evtfifo_full |
10 |
cfg_l1id |
11 |
x"b2" |
12 |
eth_rgmii_rxcount |
13 |
eth_rgmii_txcount |
14 |
eth_rgmii_status |
15 |
cfg_count_valid_packet |
16 |
cfg_count_ip_ok |
17 |
cfg_count_protocol_ok |
19:18 |
gtcrc_data |
21:20 |
gtcrc_data |
22 |
"0000">crc_charisk(1..0)>crc_charisk(1..0) |
24 |
"0000"&evt_merger_rdstate |
25 |
"000"&evt_merger_wrstate |
26 |
evt_merger_mep_seq&mep_occ |
28:27 |
eth_formatter_status |
32 |
tfc_fifo_cntin |
33 |
tfc_fifo_cntout |
34 |
tfc_fifo_occ |
35 |
derand_occ |
39:36 |
tfc_acount32 |
43:40 |
tfc_rot_count |
47:44 |
tfc_mep_count |
49:48 |
frame_count16 |
66 |
evtfifo_occ |
96 |
fragments_out |
Configuration is loaded through the Ethernet interface. The registers are not currently readable. The configuration payload also includes a 4 byte suffix containing the target ID (DNA(31..0)). If the target ID does not match the hardware, the configuration block is rejected. This is also the case if the configuration protocol type (0xf4) does not match the expected value.
| Field | Content |
| 5:0 | ethda |
| 11:6 | ethsa |
| 15:12 | ipda |
| 19:16 | ipsa |
| 20 | protocol |
| 24 | ttl |
| 30.(0) | fixed_mepda |
| 30.(1) | tfc_autogenerate |
| 30.(6) | ttc_en |
| 30.(7) | tpa_poll_en |
| 36.(5:0) | mep_evtcnt |
| 45:44 | trigger_delay |
| 46 | l1id |
| 48.(2:0) | strobe_length |
| 48.(6:4) | invert_aux |
| 48.(7) | invert_fe |
| 57:56 | latency |
| 59:58 | mep_max_fragment |
| 61:60 | mep_hwm |
| 64.(0) | mar1_cmuxen |
| 64.(1) | mar2_cmuxen |
| 64.(2) | mar1_ck40en |
| 64.(3) | mar2_ck40en |
| 64.(4) | mar1_skip_cfg |
| 64.(5) | mar2_skip_cfg |
| Description | |
| ethda | Ethernet destination address. The MAC address of the destination for the data from MAPMTFEB. |
| ethsa | Ethernet source address. The MAC address of MAPMTFEB. |
| ipda | IPv4 destination address. Should belong to a private subnet. |
| ipsa | IPv4 source address. Should belong to a private subnet. |
| protocol | IPv4 protocol number for data from MAPMTFEB (0xf2). |
| ttl | IPv4 time-to-live. A network tuning parameter. 0x40 is OK. |
| fixed_mepda | 0=Modify ethda using MEP destination info from TFC; 1=Use ethda unmodified. |
| tfc_autogenerate | 0=MEP-building commands from TFC; 1=MEP-building commands generated internally. |
| ttc_en | 0=Ignore TTC signals; 1=React to TTC signals. |
| tpa_poll_en | 0=Block Ethernet transmission; 1=Enable Ethernet transmission. |
| mep_evtcnt | When tfc_autogenerate=1, mep_evtcnt sets the number of events per MEP. Ignored when tfc_autogenerate=0. |
| trigger_delay | External triggers are delayed by this number of clock ticks. |
| l1id | The source ID written into the MEP headers. Use a unique ID for each MAPMTFEB in the set-up. Important if an event-builder is used. |
| strobe_length | MAPMTFEB integrates the incoming hits for this number of clock ticks. Small values around 2 or 3 are usually appropriate in asynchronous environments. |
| invert_aux | If asserted, the 3 LSB of invert_aux cause the corresponding AUX input to be inverted. Currently only AUX2 (SOF) and AUX3 (TRG) are affected. |
| invert_fe | If asserted, all 128 FE input signals are inverted. |
| latency | The length of the data latency pipeline. |
| mep_max_fragment | The maximum size in bytes of the data payload to be sent in a single Ethernet fragment. 1024 is safe for standard Ethernet. Values up to around 8192 are OK where jumbo frames are allowed. |
| mep_hwm | The maximum size of an IP packet is 64kByte. truncate_hwm is the value in bytes at which MAPMTFEB triggers truncation of the MEP to guarantee that it will fit in a single IP packet. Must be set low enough that there is guaranteed enough room in the currently open MEP buffer to accept any remaining queued headers for the truncated events. |
| mar[12]_cmuxen | Enable MAROC[12] charge multiplexing sequence. |
| mar[12]_ck40en | Enable MAROC[12] 40 Mhz LVDS clock. |
| mar[12]_skip_cfg | Skip serial transmission of MAROC[12] configuration sequence. |
Set-ups that do not use LHCb TFC commands should use fixed_mepda=1 and tfc_autogenerate=1.
The MEP data structure is given in the table below. A MEP can contain several events therefore the part after the MEP header may be repeated as many times as there are events in the MEP. In principle there may also be more than one bank per event but MAPMTFEB only writes one bank with type=0x09 (RICH) and version=0xc0 (TB2012).
| Field | Content |
| MEP[0] | Event index |
| MEP[1][15..0] | Event count |
| MEP[1][31..16] | Timestamp |
| EVT[0][15..0] | Event ID |
| EVT[0][31..16] | Event length |
| BANK[0][15..0] | Magic = 0xCBCB |
| BANK[0][31..0] | Length |
| BANK[1][7..0] | Type = 0x09 |
| BANK[1][15..8] | Version = 0xc0 |
| BANK[1][31..16] | Source |
| ING[0] | 0x00010000 |
| L1[0][10..0] | FE ID = DNA[10..0] |
| L1[0][15..11] | Event ID |
| L1[0][26..16] | Length |
| L1[0][31..27] | RGFMZ flags = 00101 |
| L1[1] | Timestamp |
| L1[2][15..0] | Event ID |
| L1[2][31..16] | Frame ID |
| DATA[0][7..0] | FE1[7..0] |
| DATA[0][15..8] | FE2[7..0] |
| DATA[0][31..16] | Index = 7 |
| ... | ... |
| DATA[7][7..0] | FE1[63..56] |
| DATA[7][15..8] | FE2[63..56] |
| DATA[7][31..16] | Index = 0 |
The MDF data format written by feb-mdfwriter is similar to the MEP format except that the MEP data are always separated into individual events and the MEP and MEP event headers are replace with MDF and MDF event headers. The header part is shown in the table below.
| Field | Content |
| MDF[0] | Length |
| MDF[1] | Length |
| MDF[2] | Length |
| MDF[3] | Checksum = 0x00000000 |
| MDF[4][7..0] | Compression = 0x00 |
| MDF[4][11..8] | Size = 0x7 |
| MDF[4][15..12] | Version = 0x3 |
| MDF[4][23..16] | Type |
| MDF[4][31..24] | Spare |
| EVT[0] | Mask[0] |
| EVT[1] | Mask[1] |
| EVT[2] | Mask[2] |
| EVT[3] | Mask[3] |
| EVT[4] | Run |
| EVT[5] | Orbit |
| EVT[6] | Bunch |
| Pin | 1 | 2 | 3 | 4 | 5 | 6 |
| 1 | D2<2> | D2<4> | D2<5> | D1<63> | D1<62> | D1<61> |
| 7 | D2<1> | D2<3> | D2<6> | D1<64> | D1<60> | D1<59> |
| 13 | D2<7> | D2<8> | GNDFE | GNDFE | D1<58> | D1<57> |
| 19 | D2<9> | D2<10> | MISC_IO | N/C | D1<56> | D1<55> |
| 25 | D2<11> | D2<12> | MISC_IO | N/C | D1<54> | D1<53> |
| 31 | D2<13> | D2<14> | FE2V5 | FE2V5 | D1<52> | D1<51> |
| 37 | D2<15> | D2<16> | MISC_IO | N/C | D1<50> | D1<49> |
| 43 | D2<17> | D2<18> | 3V3 | 3V3 | D1<48> | D1<47> |
| 49 | D2<19> | D2<20> | GNDD | GNDD | D1<46> | D1<45> |
| 55 | D2<21> | D2<22> | GNDFE | GNDFE | D1<44> | D1<43> |
| 61 | D2<23> | D2<24> | MISC_IO | N/C | D1<42> | D1<41> |
| 67 | D2<25> | D2<26> | MISC_IO | N/C | D1<40> | D1<39> |
| 73 | D2<27> | D2<28> | MISC_IO | N/C | D1<38> | D1<37> |
| 79 | D2<29> | D2<30> | MISC_IO | N/C | D1<36> | D1<35> |
| 85 | D2<31> | D2<32> | GNDFE | GNDFE | D1<34> | D1<33> |
| 91 | D2<34> | D2<33> | MISC_IO | N/C | D1<31> | D1<32> |
| 97 | D2<36> | D2<35> | MISC_IO | N/C | D1<29> | D1<30> |
| 103 | D2<38> | D2<37> | MISC_IO | N/C | D1<27> | D1<28> |
| 109 | D2<40> | D2<39> | MISC_IO | N/C | D1<25> | D1<26> |
| 115 | D2<42> | D2<41> | GNDFE | GNDFE | D1<23> | D1<24> |
| 121 | D2<44> | D2<43> | FE2V5 | FE2V5 | D1<21> | D1<22> |
| 127 | D2<46> | D2<45> | GNDD | GNDD | D1<19> | D1<20> |
| 133 | D2<48> | D2<47> | 3V3 | 3V3 | D1<17> | D1<18> |
| 139 | D2<50> | D2<49> | MISC_IO | N/C | D1<15> | D1<16> |
| 145 | D2<52> | D2<51> | FE2V5 | FE2V5 | D1<13> | D1<14> |
| 151 | D2<54> | D2<53> | MISC_IO | N/C | D1<11> | D1<12> |
| 157 | D2<56> | D2<55> | MISC_IO | N/C | D1<9> | D1<10> |
| 163 | D2<58> | D2<57> | GNDFE | GNDFE | D1<7> | D1<8> |
| 169 | D2<64> | D2<62> | D2<59> | D1<1> | D1<5> | D1<6> |
| 175 | D2<63> | D2<61> | D2<60> | D1<2> | D1<3> | D1<4> |