Project

General

Profile

Serial port 2 issue, RS485, RTS drops to early

Added by Fred Weiser 2 months ago

We are having a problem with serial port ttyS2 on our L138 SoM which I believe resides in the Linux driver or the TI serial port support. The port is used as a RS-485 interface running Modbus, which requires using the RTS line to control direction on the 485 transceiver. There are some cases where the RTS line is dropping before the last characters are exiting the UART serial shifter, thus causing dropped messages due to CRC failure. This error occurs only with baud rates 9600 and lower, and also seems tied to the buffer length that is being transmitted. This is not a random failure and may be consistently reproduced. All higher baud rates work fine. We are using linux kernel 3.2 built by Yocto.

The Modbus code initializes the kernel 485 feature using serial_rs485 struct; the effective code is:

struct serial_rs485 rs485conf;
memset(&rs485conf, 0, sizeof(rs485conf)); //initialize rs485 config settings
rs485conf.flags = SER_RS485_ENABLED; //enable rs485
rs485conf.flags |= SER_RS485_RTS_AFTER_SEND;
rs485conf.flags |= SER_RS485_USE_GPIO; // 1<&lt;5
rs485conf.padding[0] = 9;
rv = ioctl(port, TIOCSRS485, &rs485conf);
if (rv) { perror("unable to set IOCTL"); }

To illustrate the problem, I have included a scope shot as a file; Ch A shows the 485 output, and RX, TX, and RTS show the TTL signals from the SoM.

Any insight or suggestions would be appreciated - thanks

Picture1.jpg (237 KB) Picture1.jpg scope

Replies (14)

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Fred Weiser 2 months ago

Seems a "paste" didn't work as expected...
SER_RS485_USE_GPIO is the same as 1<<5

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Jonathan Cormier 2 months ago

The kernel driver code for RS485 support can be found here: https://support.criticallink.com/gitweb/?p=linux-davinci.git;a=blobdiff;f=drivers/tty/serial/8250.c;h=fd19e7d82137f1bcccae36b6cade7878267fc8f9;hp=eeadf1b8e093202234b30c688490da2de10a3c04;hb=685560d539688afc6bd5a735d4059c99eed24758;hpb=fcfbc9b6d9c277b2e3c4a5c63d87ef11c7babd4e

Could you try adding some debug prints and see if you can dig into what's going wrong? We are unlikely to have tested any baud rates so low so it's possible it's a long-standing bug that no one noticed before.

There does appear to be a wait_for_xmitr() call which I assume waits for the buffers to be emptied... but perhaps its not working correctly.

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Tim Iskander 2 months ago

My suspicion is that the register read in wait_for_xmitr() is "lying" and the UART is actually still serializing the data even though it says the TSR is empty.
My suggestion is to put a delay of 1 char before releasing the GPIO. This could be done (for testing) by simply putting a udelay(10^6*(2+bits)/baud) in __stop_tx() before gpio_set_value();
I don't know if this is suitable for deployed code as you are a) delaying the caller, and b) I have not checked if __stop_tx() is ever calling in an interrupt context. A more robust solution might be to
schedule a tasklet/timer function (https://linux-kernel-labs.github.io/refs/heads/master/labs/deferred_work.html) for the correct number of jiffies in the future based on baud rate. If you do that, you will obviously have to make sure your timer function doesn't flip the gpio when serial8250_start_tx() has set it.

I have not actually implemented any of this... just my thoughts!
cheers
/Tim

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Fred Weiser 2 months ago

Thanks for the ideas; I'm looking into the code cited above now; I don't have any new news there at this time.

However, here is some additional info: when the RTS line drops early, it may be a partial character, or sometimes as many as 4 characters early. It is very consistent with the same tx buffer length, and only changes when the buffer length changes. It goes something like this - at a certain buffer length, the RTS line is ok. As I keep incrementing the buffer length (and send a new packet), the RTS line falls short, and the total RTS active time remains constant. As the buffer size grows, more and more characters get cut off early due to the RTS dropping. Then at some point, the RTS active time jumps and catches up with the number of chars going out, and it is fine again (tracking the packet length).

The serial_rs485 struct permits adding time in front of the first character (delay_rts_before_send) and and time after the last character (delay_rts_after_send); however, when I tried that feature, it had no effect. That said, it may still be worth trying Tim's suggestion to see if it helps track down the root cause.

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Tim Iskander 2 months ago

It looks like (in the 3.2 kernel anyway) the only serial port driver that supports the delay_rts_after_send option is the atmel_serial.c, and it uses features of the UART to implement it.
You could implement this in the 8250 driver and set it from userspace
The that multiple characters are cut off is puzzling... the code clearly waits for both the TEMT and THRE bits to be set, which would indicate that Both the transmitter FIFO and the transmitter shift register (TSR) are empty [TEMT] AND Transmitter FIFO is empty. The last character in the FIFO has been transferred to the transmitter shift register (TSR) [THRE]. Do you see all the bytes going out on the TTL signals to transceiver? Is it possible that you are overflowing the transmit fifo and dropping characters in the driver???

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Fred Weiser 2 months ago

Please see two attached scope traces; the second with a 7 character buffer, the first with a 16 character buffer. I wanted to provide some objective evidence to your comments above... Notice the RTS active time is the same; it is that way for all buffer lengths between the two scope snapshots. You may notice a small blip on the RX (ttl) line when the RTS deasserts; this actuall occurs about 0.7 us after the RTS deasserts (I wonder if that is a switching anomaly in the 485 line driver).

If you think it may make a difference, we tie the uart 2 rts to the uart 2 cts on our baseboard, in addition to switching the 485 driver direction.

picture507-1.png (203 KB) picture507-1.png Worst case I have found (4800 baud)
picture507-2.png (197 KB) picture507-2.png At 7 chars, just beginning to fail (4800 baud)
picture507-3.png (13.7 KB) picture507-3.png Schematic snip, rts tied to cts

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Jonathan Cormier 2 months ago

What does your pinmux look like for UART2?

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Jonathan Cormier 2 months ago

Looks like wait_for_xmitr has a 10ms timeout, this might explain why the TXEN is enabled for a constant time. At low baud rates the 10ms timeout would be more of an issue I suspect.

Note in newer kernels, the stop_tx call checks the TX_EMPTY IRQ flag and if not set, enables the interrupt and exits the function. Letting the interrupt retrigger the stop_tx call when the UART is empty instead of sleeping the kernel.
https://github.com/torvalds/linux/blob/66e55ff12e7391549c4a85a7a96471dcf891cb03/drivers/tty/serial/omap-serial.c#L285

This seems like a cleaner solution though it may be easier to just increase the wait_for_xmitr timeout or scale it based on baud rate.

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Fred Weiser 2 months ago

Here's the snip:

/**
 * UART2 Pins. CTS ignored as it's floating on hw.
 */
static short baseboard_uart2_pins[] __initdata = {
/*      DA850_NUART2_CTS,
        DA850_NUART2_RTS, */
        DA850_UART2_RXD,
        DA850_UART2_TXD,
        -1,
};


Whole file may be found in https://support.criticallink.com/home/git/linux-davinci.git, branch fmc-umc, file arch/arm/mach-davinci/baseboard-usm.c

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Fred Weiser 2 months ago

Changing the timer is probably not in my best interest; my project supports down to 1200 baud with packets up to 256 in length (that's over 2 seconds). I see two possibilities:

  1. Add RTS/CTS back into the pinmux, disable the 485 config, and enable RTS/CTS flow control (this may work because I have the RTS and CTS pins tied together)
  2. Backport a newer 8250 driver

I'm still studying the current 8250 driver to understand current behavior; as the buffer length grows longer, the 485 direction control works ok, but then fails again for several buffer lengths, then starts working again.

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Jonathan Cormier 2 months ago

Fred Weiser wrote in RE: Serial port 2 issue, RS485, RTS drops to early:

Changing the timer is probably not in my best interest; my project supports down to 1200 baud with packets up to 256 in length (that's over 2 seconds). I see two possibilities:

Can you try it just to make sure we are looking at the correct code?

Note the max timeout for this will be based on the uart's max buffer size and not your message length. However, I don't recall how big that buffer is.

I think updating the driver to check the interrupt status is the way we should go. Alternatively, I did see one driver implementation using timer tasks to essentially poll for the buffer empty flag which might be easier to implement.

  1. Add RTS/CTS back into the pinmux, disable the 485 config, and enable RTS/CTS flow control (this may work because I have the RTS and CTS pins tied together)

I don't know if we've ever tried this. Wouldn't it have the same issue? Perhaps not

  1. Backport a newer 8250 driver

Backporting the whole driver is probably going to be too much work, there has been a lot of restructuring of the serial drivers. I think it would be simpler to just implement the feature/fix we are looking for. Using the newer code as a template.

I'm still studying the current 8250 driver to understand current behavior; as the buffer length grows longer, the 485 direction control works ok, but then fails again for several buffer lengths, then starts working again.

There is likely a pattern based on the hardware buffer size

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Fred Weiser about 2 months ago

I patched the 8250.c file to increase the 10ms timeout. I found the timeout is really there only to assure the uart tx fifo drains completely before shutting off the 485 driver - as such, it will never be more than about 16 characters. Increasing the timeout to 200ms should cover all my cases (essentially down to 1200 baud). The CPU burn during this polling loop is unfortunate at the slower baud rates, but this did fix the problem. I'm now evaluating if this is a tolerable fix...

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Jonathan Cormier about 2 months ago

before shutting off the 485 driver

Any idea how often the shutting down occurs?

RE: Serial port 2 issue, RS485, RTS drops to early - Added by Fred Weiser about 2 months ago

I figure it's about 10% of the CPU when operating 1200 baud and 1 response per second; it gets much lower as you approach 9600 baud (1.3%). This also takes into account the modulo 16 fifo calc assuming random packet lengths. All in all, not happy with that, but the trade-offs are worse. We will be going with this as the final solution (at least until we update the kernel to something newer). --thanks for the help!

    (1-14/14)
    Go to top
    Add picture from clipboard (Maximum size: 1 GB)