cypho's 36" DIY reef

Discussion in 'Tank Builds' started by cypho, Feb 1, 2015.

  1. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    661
    Likes Received:
    58
    Location:
    Minnetonka
    Fishies collected:
    6,676
    Just finished reading back through this thread. Fun to see how things have progressed over the last few years.

    And I realized I left out a significant contoller change. A while back (now probably over a year ago) I totally redid the html portion of my contoller with a lot of emphasis on pretty graphs using
    http://chartjs.org I enjoyed the new graphs so much I went live with web 2.0 before building on/off override buttons. Initially I planned to add them back in. But I found that it was almost as easy for me to ssh into the pi and type the override command as it was to push the button. So I still have not gotten around to rebuilding the buttons.
    [​IMG]
     
    • Like Like x 1
    • Informative Informative x 1
  2. Jonty

    Jonty I contributed!
    Lifetime Member

    Joined:
    Jul 5, 2009
    Messages:
    1,871
    Likes Received:
    83
    Fishies collected:
    14,505
    Very nice, can you give some more detail and pictures of your RO/DI setup with flow meters and TDS meter with output to the Pi?
     
  3. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    661
    Likes Received:
    58
    Location:
    Minnetonka
    Fishies collected:
    6,676
  4. eschulist

    eschulist That Office Nano Guy
    Lifetime Member

    Joined:
    Jan 31, 2012
    Messages:
    1,586
    Likes Received:
    154
    Location:
    Maplewood, MN, United States
    Fishies collected:
    12,039
    Sweet graph page. Having all of your on/off switches on that page as well would be cool.
     
  5. ikserk

    ikserk Administrator
    Staff Member Site Admin Lifetime Member

    Joined:
    Jun 9, 2013
    Messages:
    1,290
    Likes Received:
    94
    Location:
    New Hope, MN
    Fishies collected:
    9,757
    yeah, really sweet graphs. I am in the throws off using chartjs as well. but to graph data coming off the apex not a DIY controller. I have a service written already which is logging the apex data so that I'm not limited to the rolling 7 days of data that the apex keeps. care to share your front-end source code?
     
  6. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    661
    Likes Received:
    58
    Location:
    Minnetonka
    Fishies collected:
    6,676
    Sent you a PM.
     
  7. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    661
    Likes Received:
    58
    Location:
    Minnetonka
    Fishies collected:
    6,676
    The whole house smells like dead Acro today


    [​IMG][​IMG][​IMG][​IMG][​IMG][​IMG]

    Sent from my Pixel using Tapatalk
     
    #187 cypho, Nov 4, 2017
    Last edited: Nov 4, 2017
  8. Jonty

    Jonty I contributed!
    Lifetime Member

    Joined:
    Jul 5, 2009
    Messages:
    1,871
    Likes Received:
    83
    Fishies collected:
    14,505
  9. AsianReef

    AsianReef Former: 90ct
    Lifetime Member TCMAS Supporter

    Joined:
    Oct 8, 2009
    Messages:
    2,535
    Likes Received:
    62
    Fishies collected:
    22,464
    Sorry to see this. What happened?


    Sent from my iPhone using Tapatalk
     
  10. jlanger

    jlanger "The North Remembers"
    Lifetime Member

    Joined:
    May 8, 2010
    Messages:
    3,156
    Likes Received:
    361
    Location:
    New Richmond, WI
    Fishies collected:
    27,051
  11. Reefking

    Joined:
    Oct 15, 2015
    Messages:
    259
    Likes Received:
    13
    Location:
    Plymouth
    Fishies collected:
    4,807
  12. cheezybuda

    cheezybuda I contributed!
    Lifetime Member Event Committee

    Joined:
    Mar 10, 2013
    Messages:
    1,737
    Likes Received:
    110
    Location:
    SE Minneapolis
    Fishies collected:
    12,147


    Sent from my Pixel using Tapatalk
     
  13. AsianReef

    AsianReef Former: 90ct
    Lifetime Member TCMAS Supporter

    Joined:
    Oct 8, 2009
    Messages:
    2,535
    Likes Received:
    62
    Fishies collected:
    22,464
    My guess would be over heat in the tank.
     
  14. DarkSky

    DarkSky TCMAS Event Committee
    Event Committee TCMAS Member

    Joined:
    Jul 8, 2016
    Messages:
    192
    Likes Received:
    9
    Location:
    Eden Prairie
    Fishies collected:
    1,928
    Oh damn that really sucks.
     
  15. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    661
    Likes Received:
    58
    Location:
    Minnetonka
    Fishies collected:
    6,676
    A bunch of seemingly unrelated hardware failures over the last few months eventually culminated in a spectacular disaster.

    First the tip of the kalkwasser line clogged from precipitation. Pressure built up in the kalkwasser redactor until the maxi-jet mixing pump literally exploded. I eventually got everything pieced back together and working as good as ever. Too good actually (more on that later). But ca/alk were all over the place for about a month. Stressing the corals.

    Then the light on the algae turf scrubber died (from rust do to persistent high humidity). I was eventually able to get half of the light to work, but the ATS was out of commision for quite a while. Water quality suffered. Started to get hair algae in the tank - plus more stress on the corals.

    Then the sump pump died - no flow/heat for 1 night. More stress on the corals.

    To keep things alive while I waited for a new pump to arrive, I borrowed the ATS pump. So that was another week with the ATS offline. Water quality deteriorated further. Algae worse. More stress on the corals.

    New pump arrived and installed. But since this was the 4th Jebao pump to last less than a year, I switched things up and went with a different brand. I did not have the parts on hand to control the new pump, so the sump pump was set to constant speed. Controll over the pump shouldn't be critical, but it turns out that yesterday was a really bad day to not have a controllable pump. And for more ironic timing, the parts I needed arrived yesterday.

    And then came yesterday. To be continued....
     
  16. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    661
    Likes Received:
    58
    Location:
    Minnetonka
    Fishies collected:
    6,676
    Around 10 AM yesterday the controller started freaking out. Even though the outlets are normally closed (if no signal is sent they default in the off position), for some reason the controller was able to turn on outlets, but once it turned them on, it was unable to turn them back off.

    First the ATO stuck ON and started dumping RO water into the tank. In addition to the instant drop in in salinity came a drop in temperature and pH. The pH drop caused the kalkwasser reactor to kick into gear and once going, the controller was not able to shut it down, the pH shot up to 9.

    The controller noticed the ATO and kalkwasser failures and knew what to do. Shut off the sump pump to keep the bad water out of the display. The pump control system was actually working, but the sump pump was not hooked up to the controller for reasons explained in my last post. So it even though the controller wanted to shut down the sump pump, it could not.

    The controller knew exactly what the problem was, it was sending the signal to shut off the outlets, but the outlets unable to implement the command. I foresaw the possibility that the commands could get lost in transit, so after each command is sent, it checks in with the outlet and asks for a status update. If the outlet does not report the correct status, the controller will send the command again - and log the error. Although I planned for this sort of failure, I had not ever had an opportunity to test it. With 16 outlets spamming the log many times per second to complain that the commands were being ignored, the logging system crashed. So even though the controller knew there were problems I was not notified.

    The pH jump was limited by the fact that the kalkwasswer reactor only stirs itself in the afternoon (to prepare itself for a night long pH boosting session) so after the first 3 gallons it was just pumping in RO water. And the salinity did not drop too much at first either because the ATO has a safety float valve which shut the ATO down after about 5 gallons. At noon the auto-water change system came online and started pumping saltwater into the tank. And again could not stop. This actually helped things a bit - at first. But also around that time, the Iron and silica pumps came on and could not stop. A little bit of those supplements go a long way - my reservoirs held 10+years worth. I suspect up to this point, the Si and Fe were actually the most harmful thing happening.

    But then 4PM came around and the kalkwasser mixing pump kicked on and did not shut off. The tank pH shot up to 10.5
     
    #196 cypho, Nov 4, 2017
    Last edited: Nov 5, 2017
  17. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    661
    Likes Received:
    58
    Location:
    Minnetonka
    Fishies collected:
    6,676
    Any number of things would have sent me an alert right away, but since there were so many things to alert me about, it crashed and I recieved 0 alerts. I just happened to check on the tank around 2PM and I noticed the control panel was not working right. Web interface was timing out and even ssh access was sluggish.

    I troubleshooted a bit during my breaks, and eventually figured out what was going on. But by them, then it had started to snow and rush hour was kicking into full swing. I left work early, but due to the traffic I did not get home until around 6PM. The pH had been over 10 for about 2 hours, but the salinity was still OK (1.021) and at this point the corals did not look too awful.

    It is possibly things could have been salvaged at this point if I had done a 100% water change right then.

    But I'll never know. Since the water was sooo messed up, anything less than 100% would just be a waste of water. And I had no water left. Because all of my saltwater and all of my freshwater had already been pumped into the tank. I unplugged everything so at least things were not getting worse, made sure the RODI was running, loaded up the car with buckets and headed back out into the traffic to go hunt down water. I did not have the 75 gallons of new saltwater water ready to go until 8PM. So the corals had to sit in that pH 10.5 water for 4 hours.
     
  18. OP
    cypho

    cypho I contributed!
    Lifetime Member Event Committee

    Joined:
    Jan 28, 2014
    Messages:
    661
    Likes Received:
    58
    Location:
    Minnetonka
    Fishies collected:
    6,676
    Chances are that yesterdays disaster was enough to kill even a happy tank. But after more than a month of issue after issue stressing everyone out, I was not starting with a happy tank. This morning it was clear that things were not good.

    Acropora - all dead
    Pavonas and leptos - Not totally dead, but looking really bad (i'd put their odds of surviving at 5% )
    Pocilloporidae - three quarters dead, 1 quarter looks ok (so far)
    Montipora - half dead, half look OK (so far)
    Blue tang - dead
    Regal angel - MIA
    Flame angels - tattered fins but not dead (yet?)
    Snails - All dead


    The good news:

    All 4 clams seem fine!!!!
    crabs and shrimp - fine
    mandarin dragonets & starry blenny - seem fine
    Mushrooms - look fine
    psammocora - does not look too bad
    Anemones and LPS - not happy, but look OK (90% odds of surviving)
     
    #198 cypho, Nov 4, 2017
    Last edited: Nov 4, 2017
  19. eschulist

    eschulist That Office Nano Guy
    Lifetime Member

    Joined:
    Jan 31, 2012
    Messages:
    1,586
    Likes Received:
    154
    Location:
    Maplewood, MN, United States
    Fishies collected:
    12,039
    So sorry to hear Will


    Sent from my iPhone using Tapatalk
     
  20. rlinusc

    rlinusc I contributed!
    Lifetime Member

    Joined:
    Mar 8, 2015
    Messages:
    568
    Likes Received:
    20
    Location:
    Minneapolis
    Fishies collected:
    6,199
    Ugh what a nightmare- so sorry


    Sent from my iPad using Tapatalk
     

Share This Page