cypho's 36" DIY reef

Discussion in 'Tank Builds' started by cypho, Feb 1, 2015.

  1. Nickz

    Nickz Senior Member
    BOD Lifetime Member Event Committee

    Joined:
    Dec 20, 2012
    Messages:
    1,398
    Likes Received:
    239
    Location:
    Andover
    The machines are taking over.

    Holy crap that's a very bad couple of days sorry to hear this.
     
  2. Riley

    Riley I contributed!
    Lifetime Member

    Joined:
    Nov 2, 2007
    Messages:
    1,037
    Likes Received:
    131
    Location:
    Burnsville
    I'm always happy to see your thread get updated thinking "I wonder what cool thing William did now?" Your latest updates were very hard to read. I feel for you and am so sorry to see the tank went Hindenburg on you.

    Thanks for posting a detailed account of the disaster. There are things we can all learn. The club is giving you a giant group hug.
     
  3. Adam G

    Adam G Born to Reef!
    Lifetime Member

    Joined:
    Sep 28, 2007
    Messages:
    2,085
    Likes Received:
    55
    Location:
    Elko, MN
    Wow, talk about a perfect storm. So sorry to hear.
     
  4. RSnodgrass

    RSnodgrass TCMAS President
    BOD Lifetime Member Event Committee

    Joined:
    Apr 7, 2007
    Messages:
    3,737
    Likes Received:
    439
    Location:
    Lynd, MN, United States
    Wow... wow

    Well you know what kind of things I keep, not likely much you'd be interested in but if there is let me know and I'd be happy to split it once you feel up to it. I wouldn't even have a system running right now without your help so say the word.

    Your system literally DoS attacked itself, tough to ever see that one coming.
     
  5. Reefking

    Reefking Senior Member
    TCMAS Member

    Joined:
    Oct 15, 2015
    Messages:
    983
    Likes Received:
    212
    Location:
    Plymouth
    I second that motion. Let me know if I can bring you some disaster relief frags on nov 18th at the frag swap. These will all be FREE of charge! I consider you a statesman within the hobby! :);) I couldn’t even imagine how you are coping with this loss as it’s like losing a family pet. I hope you take me up on my offer...

    Regards
    Reefking
     
  6. cathorn

    cathorn Senior Member
    TCMAS Member

    Joined:
    Oct 2, 2013
    Messages:
    643
    Likes Received:
    55
    Location:
    Woodbury
    Sorry to hear Will, hope the tank gets back to normal for you.
     
  7. LoganLaBonne

    LoganLaBonne Junior Member
    TCMAS Member

    Joined:
    Aug 21, 2017
    Messages:
    49
    Likes Received:
    7
    Location:
    Minnetonka, MN, United States
    I don't have much for frags out of my little tank but PM me if you're interested in a few to get you going again. Free of course.
     
  8. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    740
    Likes Received:
    82
    Location:
    Minnetonka
    I wish I could find a defective part that I could replace and declare the controller fixed. But after a week of troubleshooting, I have been unable to find any hardware explanation for what happened. Everything seems to be working fine now. The only thing broken is my trust.

    So for now I have plugged everything back into the controller but I will be thinking hard about what (if anything) I can do prevent something like this from happening again. Possibly an second controller capable forcing all equipment into a non-deadly position (heaters off, in-tank circulation on, dosing off, etc) by cutting power upstream to things that are safer in the off position, or bypassing the main controller for things that are safer in the on position.

    For a bit of good news.

    Other than the regal angel (no longer MIA, but not alive) There have been no additional deaths. Even the pavona and leptos that I was ready to write off are beginning to recover recover. And now that there has been time for diatoms to start settling in on the dead skeleton, I can now see some of the acros are not dead - just bleached. Still a huge loss of acro, but slightly less than 100%.
     
  9. RSnodgrass

    RSnodgrass TCMAS President
    BOD Lifetime Member Event Committee

    Joined:
    Apr 7, 2007
    Messages:
    3,737
    Likes Received:
    439
    Location:
    Lynd, MN, United States
    Do I give this a thumbs up? Thanks for the update.
     
  10. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    740
    Likes Received:
    82
    Location:
    Minnetonka
    And onto other things.

    I have finally finished the hardware for my diy auto-titrator. 8 extremely precise peristaltic pumps and a reaction chamber with magnetic stir bar, RGB color sensor, pH probe, and Raspberry pi zero-w all assembled and wired up.

    I ordered chemicals for the Alk test (0.01 N HCl) today and with any luck I'll find time to get the software done by the time the chemicals arrive. If the alkalinity test works I'll order chemicals for Ca and Mg.

    Still not sure about sample size. I think it will take about 15ml to submerge the tip of the pH probe so that will be the minimum, but I can easily go much larger if needed for increased precision.




    [​IMG][​IMG]

    Sent from my Pixel using Tapatalk
     
    #210 cypho, Nov 13, 2017
    Last edited: Nov 13, 2017
    • Like Like x 1
  11. RSnodgrass

    RSnodgrass TCMAS President
    BOD Lifetime Member Event Committee

    Joined:
    Apr 7, 2007
    Messages:
    3,737
    Likes Received:
    439
    Location:
    Lynd, MN, United States
    That is worthy of a thumbs up... "The Real Good Will Hunting (cough Heaton)"!

    For continuity can you please have this "just play" a piano tune as an alarm? (o;
     
  12. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    740
    Likes Received:
    82
    Location:
    Minnetonka
    Thinking out out loud here - I am not totally sure about this math. But I think this is how the alk test will work.

    Step 1 - pump tank water into reaction chamber
    Step 2 - slowly add 0.01N HCl reagent until pH drops to 4.5

    There is no step 3. If the sample size is 10ml, the alkalinity (in meq/l) is exactly equal the amount of acid used (in ml). So if it takes 2.5ml of acid, the alkalinity is 2.5meq/l (which is 7dkH).

    My peristaltic pumps are absurdly accurate accurate - I think better than 1% of a drop, maybe even better than 0.1% of a drop, but the reagent will fall into the reaction chamber drop by drop. So unless I can somehow get the reagent into the sample without it forming into drops, the limiting factor on precision will be the size of the reagent droplets. A quick google search says that 1ml is about 20 drops.

    So with a 10ml sample, my precision is 0.05 meq/l which is 0.14 dkH. To put that into perspective, the Hana Alk Checker claims an accuracy of 5ppm which is 0.28 dkh. So a 10 ml sample could be twice as accurate as the Hana checker.

    However if I increase the sample size, each drop becomes a smaller % total and the precision goes up even further.

    With a 15ml sample, precision is 0.0333 meq/l (0.093 dkH)
    With a 100ml sample, precision is 0.005 meq/l (0.014 dkH)
    ___________________________

    It would be fun to look at graph showing minute-minute changes in alkalinity thought the day each and every day. However to see minute to minute changes, I would need to use those large 100ml samples and that would get really expensive.

    4000 ml of reagent on amazon costs $45. That is enough for 160 tests on a 100ml sample (assuming alkalinity of 2.5meq/l). So the cost is $0.28 per test. Two tests per hour = 0.28 * 48*365 = $4905.60/year. Ouch - I won't be doing that.

    On the other hand - the 10ml tests only costs $0.028. So one 10ml test per day only costs $10.22 per year. Considering I currently test alk less than once a week - that is still a huge step up from where I am at now.
    ___________________________

    However, even if I want to try to maintain Alk rock solid by adjusting dosing rate over the course of the day, I don't think I there is really any need to run super accurate tests all day long every day.

    Because even if the total consumption goes up and down a bit from day to day, I think it is safe to assume that % of daily consumption used at any particular minute would not change all that much from day to day.

    First I could build a minute-to-minute consumption profile by frequently measuring Alk consumption using large samples over the course of one day. Then going forward I should be able run one large sample test per day and use that to scale the profile to estimate consumption at any given time.

    Then for the rest of the day I could do periodic small sample tests for an early alert in case something goes wrong. Wouldn't that be fun a fun emergency alert. "Warning - Alk is 0.3 dKh above expected. Running large sample test to verify - please stay tuned."
     
    #212 cypho, Nov 13, 2017
    Last edited: Nov 13, 2017
    • Informative Informative x 1
  13. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    740
    Likes Received:
    82
    Location:
    Minnetonka
    Still no more controller crashes. I have been reviewing all of the code looking for a software cause. And I think I found the problem:

    The outlets are controlled in sets of 8. So to turn on/off an one outlet:
    1. Controller sends request for the status of the 8 outlets in the set
    2. Controller then sends a command to the set status of all 8 outlets (only one is different but the command includes the status of all 8)
    3. Controller sends request for the status of the 8 outlets to make sure the change was successful
    4. If change was not successful report error and try again
    When only changing one outlet at a time, that works perfectly. The time between step 1 and step 3 is typically a tiny fraction of a second. A really tiny fraction - I think somewhere around 1/100,000 of a second. But if the controller tries to change two outlets at exactly the same time, it is possible for this to happen.
    1. Controller sends request for the status of the 8 outlets in the set on behalf of outlet A
    2. Controller sends request for the status of the 8 outlets in the set on behalf of outlet B
    3. Controller sends a command to the set status of all 8 outlets on behalf of outlet A (initial status with only A changed)
    4. Controller sends a command to the set status of all 8 outlets on behalf of outlet B (initial status with only B changed)
    5. Controller sends request for the status of the 8 outlets in the set to see if change A was successful (this will return an error because B overwrote the changes for A)
    6. So outlet A will send the command again.
    7. Controller sends request for the status of the 8 outlets in the set to see if change B was successful (this will return an error because A overwrote the changes for B)
    8. So outlet B will send the command again
    9. Return to step 5 - continue loop forever
    The chances of that happening are so miniscule it never occurred to consider it a problem. But I it seems to match the symptoms pretty well.

    For a solution, it is easy to spot a collision, but the controller software is multi-threaded so it would not be so easy to prevent a collision. I don't think the fish will care if once a year the kalk pump stays on for 1/100 of a second longer than planned. So I think I will just let the collisions happen and then wait a random fraction of a second before trying again so it cannot get stuck in a never-ending loop of collisions.
     
    #213 cypho, Nov 21, 2017
    Last edited: Nov 21, 2017
  14. ikserk

    ikserk Administrator
    Site Admin Lifetime Member

    Joined:
    Jun 9, 2013
    Messages:
    1,681
    Likes Received:
    256
    Location:
    New Hope, MN
    yay for race conditions!
     
  15. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    740
    Likes Received:
    82
    Location:
    Minnetonka
    Back to the auto-titrator.

    I missed my self-imposed deadline. The chemicals arrived but I have not finished the software. But I have made progress.

    Raspberry pi's are very powerful computers, but they are not great at things that require perfect timing and they have a limited number of IO ports. The dosing pumps use stepper motors which are extremely precise, but are somewhat more complicated to control. I could have controlled the motors directly from the pi, but I might not have been able to control as many motors or with the level of precision that I wanted.

    So my plan was to built stepper motor controllers out of arduinos which are relatively primitive compared to the pi, but they are better suited for this task. Each arduino will control two pumps and then the raspberry pi could send a simple command like "hey pump D, please rotate 1000 steps clockwise at 10 rpm, let me know when you are done" and the arduinos would deal with all of the intricacies making that happen.

    I wrote the arduino software required to control the stepper motors a long time ago. The next step was to get the arduinos talking to the pi. It was actually harder than I expected. Although arduino and raspberry pi both have i2c libraries available they are apparently slightly different and do not like each other. Many things that should work, did not and it was really hard to figure out where the problem was. I am sure it did not help that this is my first arduino project.

    But I am happy to report that I have it working and the pi is able to get the 8 dosing pumps to do whatever it wants.

    With one caveat. The power supply I am using does not put out enough juice to turn all 8 motors at the same time. I may eventually switch to a bigger power supply, but but I don't think that is actually an issue because I don't think I will ever actually need to use more than one or two pumps at any one time.

    Next task - calibrating the pumps and programming the titration routine.
     
  16. RSnodgrass

    RSnodgrass TCMAS President
    BOD Lifetime Member Event Committee

    Joined:
    Apr 7, 2007
    Messages:
    3,737
    Likes Received:
    439
    Location:
    Lynd, MN, United States
    I realize I'm taking a break from work to read this but I feel like I'm still working... does that count? (o:

    Good translation to English though! Excited to see it coming together.
     
  17. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    740
    Likes Received:
    82
    Location:
    Minnetonka
    It might be working!

    Code:
    pi@ReefZero:~ $ ./test.py
    Starting prime and drain.
       Done
       It took 144.352033138 seconds to prime and drain.
    Starting rinse.
       Done
       It took 143.117043972 seconds to rinse.
    Starting Titration.
       Adding 10 ml sample and 2.0 ml HCl
       Quickly titrating to pH of 5.0
       Titrating to pH of 4.6
       meq/l		 pH
       2.58245046106 4.68826629681
       2.58858151854 4.59962331813
       Slowly titrating to pH of 4.5
       meq/l		 pH
       2.59164704728 4.57331795592
       2.59471257603 4.5769498009
       2.59777810477 4.55402095663
       2.60084363351 4.56335336724
       2.60390916225 4.53715123126
       2.60697469099 4.53613560401
       2.61004021974 4.52227461859
       2.61310574848 4.52227461859
       2.61617127722 4.52227461859
       2.61923680596 4.51554151243
       2.62230233471 4.4966983498
       Done
       It took 487.101300955 seconds to titrate.
    
    The alkalinity is 2.62176520069 meq/l
    
    Total Test time:  13.3464886824 minutes.
    
    pi@ReefZero:~ $ ./test.py
    Starting prime and drain.
       It took 144.355286121 seconds to prime and drain.
    Starting rinse.
       Done
       It took 143.084019899 seconds to rinse.
    Starting Titration.
       Adding 10 ml sample and 2.0 ml HCl
       Quickly titrating to pH of 5.0
       Titrating to pH of 4.6
       meq/l		 pH
       2.58245046106 4.68826629681
       2.58858151854 4.64411045265
       2.59471257603 4.6052704577
       2.60084363351 4.59395663858
       Slowly titrating to pH of 4.5
       meq/l		 pH
       2.60390916225 4.56377253814
       2.60697469099 4.57379414456
       2.61004021974 4.54110485735
       2.61310574848 4.54176352008
       2.61617127722 4.53563603664
       2.61923680596 4.52227461859
       2.62230233471 4.52227461859
       2.62536786345 4.51221865343
       2.62843339219 4.49677589693
       Done
       It took 476.826489925 seconds to titrate.
    
    The alkalinity is 2.62779337817 meq/l
    
    Total Test time:  13.1747404178 minutes.
    
    pi@ReefZero:~ $ ./test.py
    Starting prime and drain.
       Done
       It took 144.357420921 seconds to prime and drain.
    Starting rinse.
       Done
       It took 143.088948965 seconds to rinse.
    Starting Titration.
       Adding 10 ml sample and 2.0 ml HCl
       Quickly titrating to pH of 5.0
       Titrating to pH of 4.6
       meq/l		 pH
       2.58245046106 4.68826629681
       2.58858151854 4.62703156186
       2.59471257603 4.59403328119
       Slowly titrating to pH of 4.5
       meq/l		 pH
       2.59777810477 4.58356128449
       2.60084363351 4.56377253814
       2.60390916225 4.56880000684
       2.60697469099 4.53881104657
       2.61004021974 4.54863254513
       2.61310574848 4.52227461859
       2.61617127722 4.53248442419
       2.61923680596 4.52227461859
       2.62230233471 4.50847003683
       2.62536786345 4.49427400216
       Done
       It took 466.508852005 seconds to titrate.
    
    The alkalinity is 2.62413137654 meq/l
    
    Total Test time:  13.002863582 minutes.
    
    
     
    • Like Like x 2
  18. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    740
    Likes Received:
    82
    Location:
    Minnetonka
    2.621 meq/l
    2.627 meq/l
    2.624 meq/l

    Wow - that is some ridiculous precision. Way better than I expected.

    It turns out the theoretical precision is not limited by the size of a water droplet. I am using a small pipet tip on the end of the hoses. If I pump the acid in slowly it does form a big water droplets that would limit the precision. But if I have the acid pump go full speed for a fraction of a second and then abruptly stop, instead of forming a droplet, it just squirts a tiny bit of acid into the vessel.

    I am currently using 250 step pulses of acid between pH readings. Now a step is a meaningless unit so let's put that into perspective. When I calibrated the pump I came up with 81,552 steps in 1 ml. So instead of 20 drops per ml, that works out to 326 pulses per ml. That makes the theoretical precision on a 10ml sample, 0.003 meq/l or 0.008 dKH!

    I'm not saying my actual results will be that good. It just means the sample size and water drop size are not going to be the limiting factor. The limiting factor is going to be the pH probe.

    Perhaps I should buy a new probe - or at least re-calibrate it. (probe is a year old and has not ever been re-calibrated)
     
    #218 cypho, Nov 25, 2017
    Last edited: Nov 25, 2017
    • Like Like x 2
  19. DarkSky

    DarkSky I contributed!
    Lifetime Member Event Committee

    Joined:
    Jul 8, 2016
    Messages:
    744
    Likes Received:
    197
    Location:
    Eden Prairie
    Looks precise! Consider doing one for Ca/Mg?

    Edit: Nevermind, just saw your post about ordering reagents for the other two tests!
     
  20. RSnodgrass

    RSnodgrass TCMAS President
    BOD Lifetime Member Event Committee

    Joined:
    Apr 7, 2007
    Messages:
    3,737
    Likes Received:
    439
    Location:
    Lynd, MN, United States
    Well I have to say that's a nice surprise not having it form droplets... I can't say I would have expected anything but that.
     

Share This Page