FTDI Community

Please login or register.

Login with username, password and session length.
Advanced Search  

News:

Welcome to the FTDI Community!

Please read our Welcome Note

Technical Support enquires
please contact the team
@ FTDI Support


New Bridgetek Community is now open

Please note that we have created the Bridgetek Community to discuss all Bridgetek products e.g. EVE, MCU.

Please follow this link and create a new user account to get started.

Bridgetek Community

Author Topic: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi  (Read 35196 times)

Miriam

  • Newbie
  • *
  • Posts: 3
    • View Profile

Hi there,

I am new to this community so I hope I chose the right subforum to post.

I have an FTDI USB device that sends a huge amount of (known) data via the FTD2XX driver in 245-FIFO-mode. I wrote a little C code that reads the data and checks it for correctness. With every call of FT_Read(), an array of 256 bytes is processed.

When I run this code on my Windows desktop, it works fine. However, when I cross compile and run the executable on a Raspberry Pi, it results in data loss. I built a delay of 2 seconds in my code after the device starts sending data and before the first call of FT_Read(). Since that, the fault in the read data always occurs at the same byte.

What is the reason of this data loss (buffer overflow?) and how can I prevent it?

I use a Raspberry Pi 3 with the latest version of Raspbian OS Lite. The code is cross compiled on a Windows 32bit workstation with GCC. The libraries used are those packed with the libftd2xx-arm-v6-hf-1.4.6 driver I downloaded from the official FTDI website.

Thank you for any ideas.
Logged

FTDI Community

  • Administrator
  • Hero Member
  • *****
  • Posts: 892
    • View Profile
Re: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi
« Reply #1 on: December 05, 2017, 01:57:37 PM »

Hello,

Thank you for your post.

The following application note covers FIFO Basics and should be of use to you:
http://www.ftdichip.com/Support/Documents/TechnicalNotes/TN_167_FIFO_Basics.pdf

Have you included the Buffer Flags in your hardware design? These should help to mitigate data loss.

Does the behaviour change if you remove the 2 second delay or compile the code natively on the Raspberry Pi?

Best Regards,

FTDI Community
Logged

Miriam

  • Newbie
  • *
  • Posts: 3
    • View Profile
Re: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi
« Reply #2 on: December 05, 2017, 03:38:02 PM »

Thanks for your answer.

Yes, the device uses buffer flags for control issues. Proof of that is that I can run the same C code on Windows with the very same device connected to the Windows PC (or almost the same code, as FT_SetVIDPID() is neither defined nor necessary).

The 2 seconds delay is just to grant that the fault occurs at the same byte. Before I implemented this, the error started somewhere between the 1500th and 2100th call of FT_Read, i.e. some time after 1500*256 and before 2100*256 bytes are read.

I did not compile on the Pi before posting here. Now I did and it shows the same result  :(

Any other ideas? I also can post code if this helps.
Logged

Miriam

  • Newbie
  • *
  • Posts: 3
    • View Profile
Re: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi
« Reply #3 on: December 11, 2017, 03:51:57 PM »

Still no solution found...

Is there a way to check if buffer flags are activated in the D2XX ARM driver? The hardware design obviously supports these (as mentioned before, the test runs on a Windows workstation).

Or shall I contact FTDI support via E-Mail?
Logged

FTDI Community

  • Administrator
  • Hero Member
  • *****
  • Posts: 892
    • View Profile
Re: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi
« Reply #4 on: December 11, 2017, 04:57:33 PM »

Hello,

I believe that the Buffer Flags are set internally by the IC and thus cannot be queried by the driver.
It would be best to check these with an oscilloscope, its worth noting the RD# or WR# strobe inputs must be throttled when the TXE# or RXF# buffer flags stay high for over 400 nSec.

Does you code implement the FT_GetQueueStatus function before performing the FT_Read's? An example of this can be found in the D2XX Programmers Guide:
http://www.ftdichip.com/Support/Documents/ProgramGuides/D2XX_Programmer's_Guide(FT_000071).pdf

Best Regards,

FTDI Community.
Logged

DaveHarper

  • Newbie
  • *
  • Posts: 4
    • View Profile
Re: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi
« Reply #5 on: February 19, 2018, 02:37:09 PM »

Miriam -

I realize that this thread is now over two months old but I'm wondering if any progress has been made on this issue?  The reason I ask is that I'm facing a similar problem - data loss/corruption originating on a board with an FT232H operating in 245 FIFO mode (async) and connected to a Raspberry Pi (3B).  Like you, my application on the Pi is written in C, uses the D2XX library (arm-v6-hf-1.4.6) and is compiled with GCC.  Also like you, the code runs fine on a Windows 7 64-bit system.

My application is a data logger and the hardware consists of a small board which takes as input a 30MHz Manchester encoded bit stream.  A CPLD decodes this bit stream back into 8-bit data and strobes it into the FT232H (and yes, the CPLD monitors the TXE# line to prevent overruns).  The data then runs over a short USB cable to the Pi which accumulates it and writes it out to an attached 1TB USB drive.  Each file accumulates roughly 10MB before it is closed and the next one opened.  For troubleshooting I have a logic analyzer monitoring the 8 data lines and WR# between the CPLD and the FT232H and a hardware USB capture unit between the data logger board and the Pi.  Thus, I can monitor the data between the CPLD and the FT232H and between the FT232H and the Pi.  This, along with the output file, allows me to monitor data from where it's originally generated to where it's finally stored.

After realizing that this seemed to be a Pi only related problem, I conducted a series of tests.  In the first one I reprogrammed the CPLD to simply generate a continuously incrementing byte pattern.  This was written into the FT232H at a maximum rate of 7.5MHz, pausing only to wait on TXE# if needed.  The Pi code consisted of a loop which did an FT_Read() for 64KB and then checked the data.  Not everything was checked - only the first and last bytes in each buffer.  If there was any data loss, it would have to be in multiples of 256 bytes to avoid detection.  This test successfully transferred over 15GB of data without any detected errors before I stopped it.  The actual throughput on this turned out to be closer to 7MB/sec based on the amount of data received by the application and the time over which the test ran.

The next test extended the first test to write the incoming data to disk in 10MB files.  The Pi program was modified to do this in two threads - the input thread which accumulated the data and an output thread which wrote this to disk.  This was necessary as the fwrite() can become lengthy if the internal buffers fill up and must be flushed to disk.  I found that I was able to sustain just over 2MB/sec with this configuration.  This test ran for several hours before I manually stopped it.  Had the input thread detected an error it would have aborted the test immediately.  Further, I wrote a small script to check the output files and no corruption or dropped data was detected in any of them.

Next, I performed a final test where the CPLD was reprogrammed back to the original Manchester decoding configuration and, instead of running continuously, only a 4MB sample of data would be accumulated from the bit stream and written to disk.  This size was chosen because the depth of my logic analyzer is only 4MB and it allowed me to capture all of the decoded data generated by the CPLD.  This past weekend I ran over a half dozen tests.  The data rate of the actual logging data is only around 100KB/sec so each test only took about 40 seconds.  Each disk file was found to contain anywhere from 1 to 7 errors (one file actually made it all the way through without errors) so, while the test itself ran quickly, analysis of logic analyzer, USB and disk files could take hours to determine exactly what had happened and where, so this limited the number of tests run.

What I found was that the errors could be summarized in three categories.  The first was the injection of the byte pattern "0x32 0x00" into the data stored on disk.  While this was not present in the logic analyzer trace, it was present in the USB analyzer files.  Each high-speed data packet typically consisted of 512 bytes of data.  The first two were "0x32 0x00" followed by 510 bytes of data that were generated by the CPLD.  Somehow the FT232H and D2XX library seem to be getting out of sync such that this 2-byte header ends up in the data being returned from the FT_Read().  The second catagory consisted of dropped data, anywhere from a few dozen bytes to over 1KB, where the data from the CPLD was also seen to correctly cross the USB and subsequently be dropped somewhere between the USB port on the Pi and the appliction.  The final category also consists dropped data but in this case it was dropped in the FT232H as the data from the CPLD was NOT seen on the USB trace.

I don't know if any of this information is helpful to you or if you have already resolved your issue.  In my case, my final application absolutely requires a data logger and I have spent a lot of time getting this far.  I'm really hoping this can be resolved, otherwise I'm faced with starting over from scratch and developing a board that does not use USB to transfer the data from the logger board to the Raspberry Pi.

Regards,
Dave
Logged

FTDI Community

  • Administrator
  • Hero Member
  • *****
  • Posts: 892
    • View Profile
Re: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi
« Reply #6 on: February 20, 2018, 11:52:46 AM »

Hello,

There are known issues with FIFO in the Linux d2xx driver, unfortunately there are currently no plans to fix these.

However, one of our Linux customers has used a third party FTDI library called LibFTDI to get 40 MByte/sec performance with Sync FIFO with the latest Linux kernels.
It may be worth giving this library a try to see if it resolves your issues.
 
http://www.intra2net.com/en/developer/libftdi/
 
Keep in mind FTDI cannot support this API since it is developed and maintained by a third party.


Best Regards,

FTDI Community
Logged

DaveHarper

  • Newbie
  • *
  • Posts: 4
    • View Profile
Re: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi
« Reply #7 on: February 20, 2018, 12:08:41 PM »

Thank you for the tip on the alternative driver.  I will try it today and, once I've had a chance to test it, will post the results of whether or not it worked for my application.

Dave
Logged

DaveHarper

  • Newbie
  • *
  • Posts: 4
    • View Profile
Re: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi
« Reply #8 on: February 25, 2018, 08:11:18 PM »

There is a fairly steep learning curve in switching to libftdi (which is built on top of libusb, both of which must be understood), but I've performed enough preliminary tests at this point that I am confident that this approach will work for me.  Thanks for your help in pointing this out.

Dave
Logged

EAD

  • Newbie
  • *
  • Posts: 1
    • View Profile
Re: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi
« Reply #9 on: December 20, 2018, 07:54:22 AM »

Hello Dave,

The problems I have encountered are very similar to yours
I have tried to use third party driver(libftdi) but I  can not overcome!
I wonder if you can fix your problems and if so, How?
Did you encounter any problem while integrating driver?
Could you please give some information ?

Best Regards,

EAD
Logged

DaveHarper

  • Newbie
  • *
  • Posts: 4
    • View Profile
Re: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi
« Reply #10 on: December 21, 2018, 04:07:17 AM »

Could you please give some information ?

I have responded to your email address.  If you did not receive it, please repost here.

Dave
Logged

Roy

  • Newbie
  • *
  • Posts: 1
    • View Profile
Re: Reading data from FT232h via FT2XX driver - data loss on Raspberry Pi
« Reply #11 on: August 15, 2019, 01:47:48 PM »

Hello Dave

Have you solved your problem? We have a similar problem with our application and I wondered if you have already found the cause and a solution.

Roy
Logged