thesolarpowerexpert.com

The Solar Power Expert Blog

An Open Source High Speed Data Logger

by Chris - August 23rd, 2011.
Filed under: Charge Controller. Tagged as: , , , , , , .


I’ve been eye-balling the Leaf Labs Maple board recently. It has several attractive features and I’ve been thinking it would be a good fit for use with the open source free charge controller project.

Among it’s many features, the Maple sports a 32-bit ARM microcontroller, is Arduino compatible in terms of both hardware and software, and uses open source, royalty free development tools. Like the Arduino, it programs via a USB port and all the development software installed and ran flawlessly on my Aspire netbook runing an Intel Atom N450 processor. My experience playing with this neat little device has been suberb. Everything installed and ran flawlessly.

The Wiring language used to program the board was very quick to pick up. The language is just a light wrapper that assists people to get strait to coding without having to worry about all the device specific operations that strait C coding requires.


  • Creating a Data Logger Hardware

    I needed to run a test at work that required a high speed data logger. Since we didn’t have the budget to purchase one, I decided I’d give a shot at quickly building one out of a Maple and this SD Card Shield from Seeed Studios. Although it was origonally developed for the Arduino, it plugs right onto the Maple (I’m using Rev 5, by the way) since it’s pin compatible.

    Voilà! I now have a high speed data logger! All I need now is some software to make it work.


  • Create the Data Logger Software

    Creating the software wasn’t as easy as putting the hardware together, but it was still pretty easy. Thanks to Leaf Labs awesome forum and the extensive documentation, it was largely an excersize in downloading open source code and making everything play ‘nice’. I’ll go through each stage of the development for the sake of posterity.


    • Implementing a Fixed Point Math Library

      For me, an attractive feature of using the Maple is that it uses a 32-bit processor. This means that I can use native 32-bit integers and implement a fixed point math library to implement math-heavy algorithms such as digital signal processing (DSP). It’s not as accurate using a floating-point processor, but it’s orders of magnitude better than trying to use an 8-bit processor. So for me, the first order of business was implementing a floating point math library.

      I wasn’t the only one to lock onto this major feature of the Maple. This Fixed Point Math Library thread in the forum pointed me at the libfixmath library as well as forum user Crenn‘s benchmarking code.

      The trick to getting the library working was to drop the libfixmath folder into the libraries folder of the Maple IDE installation directory.


    • Combining the fix16 Math Library with the Analog to Digital Converter (ADC)

      Next, I needed to read a value from the ADC and convert it to a voltage using the fix16 math library. This ADC_test.pdf program I wrote did just that. Next, I needed to be able to write these values to the SD card in order to create a full data logger.



    • Writing ADC Data to the SD Card

      In order to write data to the SD card, I needed a source code to enable the Maple to talk with the SD card on the shield. Going back to the forum, I found this thread which pointed me to the SDFat Library for the Maple.

      Again, implementing the library into the IDE required dropping the mapleSDfat folder into the libraries of the Maple IDE installation directory and rebooting the IDE. A couple hours of tweaking and I was able to produce this ADC_SD_Write.pde program, which now lets me read an ADC value, convert it to a voltage, and write that value to the SD card once per second.


    • Speeding Up the Data Logger

      At this point I was really close! For the test I wanted to run, I needed to sample 3 channels as fast possible. Since I’ve developed data-logger like for other hardware platforms, I know from experience that the fastest way to operate is to create two FIFO buffers in memory. I can then implement an interrupt handler that is called at a set amount of time to read the ADC and drop the data into the FIFO. Once the first FIFO fills up, I point the interrupt handler at the other FIFO while I write the first FIFO to the SD card in the loop() part of the program.

      Ideally, I’d like to sample each channel at rate of 1000 samples per second (1 ksps). That means I need to create an interrupt vector that is services 1000 times per second, or 1 millisecond. This Timer_Interrupt_test.pde program does just that.

    Data Logger Built From Leaf Labs Maple

    My completed data logger. Three inputs, one amplified by an op-amp, is read by the Maple at 300 sps and written to the SD card as a CSV file.


    • Putting It All Together

      After much fiddling, I wasn’t able to achieve a rate of 1 ksps with 3 ADC channels. The best I could do was 300 samples per second. Still, that’s really fast and lets me sample signals up to 150 Hz (see Nyquist sampling theorem). The data logging code is available in this program: Data_Logger.pde

      I could speed up my sample rate significantly if I could write data to the SD card faster. The current program stores data as ASCII text in a CSV file. If I store data on the SD card as binary data, I could easily achieve a sampling rate of 1 ksps for 3 channels. That will be my future work.


  • Development Computer

    As I said at the beginning, I’m doing all my development on an Aspire One netbook with an N450 processor. Just like the one below. It’s got an awesome battery life and lets me program for hours, wherever I happen to be at – home, the coffee shop, or my boat.






16 Responses to An Open Source High Speed Data Logger

  1. Dear Chris,
    very nice work. Keeping your work as an example, I’m trying to use the sdfat library for a project of mine with a maple-compatible STM32 board, but I’m experiencing some problems, since sdfat is expecting to access standard avr libraries (like avr/pgmspace.h and avr/io.h) within maple ide. How did you manage to solve that?
    Thanks,
    Luca

  2. Hey Luca,

    When you unpack the Maple SDFat Library to the ‘libraries’ directory of the IDE installation, look inside the directory. On my computer its called ‘mapleSDfat’. Inside is an ‘examples’ directory and inside is another directory called ‘FileSys’. That is the example program that worked on my Maple and SD-shield. I used that as a template for getting my own code going.

    Cheers!

    Chris Troutner

  3. Dear Chris,
    thanks a lot. I manage to solve: I was actually loading the wrong library.
    Best Regards

    Luca Schenato

  4. Hello there.

    I manage to get it to compile and run, but it seems like I have a problem with the SPI-pins.

    My card is an Olimexino STM32.

    Any ideas to help?

    Luca: What card are you using?

    /Christian

  5. Hey Christian,

    My first instinct would be to check your schematic. It sounds like you might have a pin-out discrepancy between the two boards. If you’re pretty sure the pin-out is correct, try lowering the frequency of the SPI port. It may be trying to communicate with the SD card too quickly.

    Cheers!

    Chris Troutner

  6. Hello Chris
    I’m sure it’s corretct. But then I have been sure before, and still been wrong…

    If I can get this going I highly recomend this card. Cheap and well built.

    I’ll be back.

    //Christian

  7. Hi again.

    Got a bit on my way today.

    You were right about the pins.

    #define BOARD_SPI2_NSS_PIN 25
    made the trick at line 61 in maple.h.

    Also I had to do:
    HardwareSPI spi(2);
    instead of spi(1)

    Now I can open a file and read it, writing is for another day. Now my bed is calling.

    Thanks.

    /Christian Nilsson
    ———————
    +46 703 190 190
    mail: chin@chin.se
    http://chin.se
    ———————

  8. Hello All.

    Thought I’d share my experience now that it all works.

    In order to get it going I had to do the following:
    In the file maple.h (line 61 or there about) I had to change a pin. Should be 25 for the nss, or CS as it’s called too.
    Here is where the file is located in a mac installation:/Applications/MapleIDE.app/Contents/Resources/Java/hardware/leaflabs/cores/maple/maple.h

    When I initiate the spi I have to tell it to use the second spi-port in my sketch:
    HardwareSPI spi(2);

    Now it works.

    The card I’m using from Olimex, is called Olimexino-stm32.

    //Christian

  9. Thanks for the updates Christian!

    Olimex makes some good products. I’m happy you were able to get this code working on that platform too.

  10. Hey,
    I’m using the Olimexino STM32 and a get the following feedback from the IDE:

    Going to build using ‘armcompiler’ (ARM)
    Compiling core…
    Compiling libraries: SdFat, libfixmath

    /Applications/MapleIDE.app/Contents/Resources/Java/libraries/SdFat/Sd2Card.cpp: In member function ‘uint8_t Sd2Card::readData(uint32_t, uint16_t, uint16_t, uint8_t*)':

    /Applications/MapleIDE.app/Contents/Resources/Java/libraries/SdFat/SdFile.cpp: In member function ‘uint8_t SdFile::rmRfStar()':

    Compiling the sketch…

    In function ‘void loop()':

    Linking…
    Computing sketch size…

    /var/folders/h4/tjgpg3w91f7052hngg4y1_n00000gn/T/build3908845637714798785.tmp/SD_Write.cpp.bin :
    section size addr
    .data 36016 0
    Total 36016

    Binary sketch size is reported above. Check it against a 108000 byte maximum.
    Loading via dfu-util
    Resetting to bootloader via DTR pulse
    Searching for DFU device [1EAF:0003]…
    Found it!

    Opening USB Device 0x1eaf:0x0003…
    Found Runtime: [0x1eaf:0x0003] devnum=0, cfg=0, intf=0, alt=1, name=”DFU Program FLASH 0x08005000″
    Claiming USB DFU Interface…
    Setting Alternate Setting …
    Determining device status: state = dfuIDLE, status = 0
    dfuIDLE, continuing
    Transfer Size = 0x0400
    bytes_per_hash=720
    Starting download: [##################################################] finished!
    state(8) = dfuMANIFEST-WAIT-RESET, status(0) = No error condition is present
    Done!
    Resetting USB to switch back to runtime mode

    What do this “In member function” mean, and why are the red…

  11. I get that same warning on my Maple compiler. I don’t know why they are red or why they show up. However, the device is programmed and running, so who cares, right?

  12. After all it is running now. At the moment I posted this it did not. However, it still gives the warning messages, I gonna try to figure that out.

    The problem they is did not run what probably the 8GB SD card FAT32, now i use a 2GB FAT16… Gonna sort out is fat is the limitation, or that is is just a bug.

  13. Hi Chris,

    Does this mean that 3 channels enabled is 300 samples/sec aggregate? So just enabling one channel would be 1000 samples/sec? Would this code work with the Olimexino stm32 with the built in micro sd port?

    Thanks…

  14. Sort of. I needed 3 channels and the highest thoroughput I could achieve was 300 sps for each channel. In theory however, you should be able to hack the code at achieve 1000 sps for one channel. The limiting factor is the overhead required by the processor to shuffle data around.

  15. I report here my experience with “maple-sdfat” library.

    At the beginning I downloaded the file SdFat.zip from here http://code.google.com/p/maple-sdfat/downloads/list all seems to work in the sense that I was able to compile and download the code in the mcu. But… the sketch was not able to read any file and it stops with this error “volume.init failed”. The problem was in the file FatStructs.h for this reason I suggest you to start from the mapleSDFat.zip archives (svn repository are ok also.

    Second point: the source file Sd2Card.cpp manages the chip select of SD card; for this reason it is necessary to adapt the digitalWrite() and pinMode() functions to the hardware you are using. For the “Olimexino” the I/O is number 25 instead of 10. The maple.h mod is correct but has not effect in this case.
    I’m using 4GB FAT32 SD card and seem to be ok.

  16. Hello
    is it possible to fix this error ?

    (IN RED)
    /home/Maple/maple-ide-v0.0.12/libraries/mapleSDfat/Sd2Card.cpp: In member function ‘uint8_t Sd2Card::readData(uint32_t, uint16_t, uint16_t, uint8_t*)':

    (IN RED)
    /home/Maple/maple-ide-v0.0.12/libraries/mapleSDfat/SdFile.cpp: In member function ‘uint8_t SdFile::rmRfStar()':

    Thank you

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>