These Poolish Games

Test Dive #5 is in the books.  

We set up an obstacle course underwater on Sunday to attempt to navigate and put ROV in the water.

Obstacle course sighted!

  We started flying, and immediately noticed that our balance in the water was way off, so we started working to adjust our weight distribution to get the ballast correct.  Turns out that split-shot sinkers from a fishing tackle box worked perfectly for this!

They have upgraded from catching sunfish in the river to ballasting an ROV - a huge promotion in a fishing weight's life, I'd think.  

Adjusting the weights 

Back in the water we went.  

Thanks to David H. for hosting us at his apartment pool again!

Some of the team chats with David while rebooting the ROV

Here's the part where things went wrong.  One of our pre-flight checks on our checklist (yes, we print out flight checklists every time we dive...it is important to be thorough!) is to check the seals on the tubes.  This is important because apparently you don't want water inside the main electronics tube of your ROV, who knew?  But what we didn't do was check the seals again after adjusting all the weights on the ROV.  So although newly balanced, we sprang a leak.  I think it came loose when we moved the tube around adjusting the weights.

So yes, you are correct, that is BAD NEWS BEARS.  

Bleh...water inside

It went down something like this.  I (James) was at the controls, wondering why we couldn't fly straight, just looping in circles.  Danny and Richard and Arthur were calling out to me to stop spinning in circles.  Someone noticed that there was fog in the main tube.  We pulled the ROV out of the pool and...well, yep, it's now about 5% full of water, enough to puddle up in the corner.  

The good news is that we never lost the video feed or the Cockpit from the ROV, so it still works.  The bad news is that obviously our dive was over for the day.

Drying off

Since we were gathered up and Arthur had brought some pro video equipment, we recorded some interviews we hope to incorporate later into a video we're making about the entire experience.  

The Interview that North Korea didn't want you to see

It takes a lot of gear to do things our way

Some positives we gain from this dive are that we had more control time underwater than ever before, thanks to our new LiPo batteries.  Next steps are to fix our trim issues so that we can fly straighter underwater, and to attempt the obstacle course again.  

austinrov documentary teaser

None of us are filmmakers today but we're learning. The team worked on our interview questions and filming techniques today. Lots of footage was captured. Here we present a teaser, only a very small taste of the fun we had.

AustinROV teaser from Daniel Epperson on Vimeo.

We verified that the new batteries and camera servo were working as expected. One of the ESCs lost programming but it wasn't noticed or fixed. That might come back to haunt us later however it will be an easy fix.

Some footage for your enjoyment

This past Friday, Danny E and I went to Spring Lake to scout out a potential test dive in a real world environment.  Unfortunately, the vegetation level and shallowness of the lake doesn't look great for using the ROV there, but we still managed to have a productive day.  We pieced together a bunch of the photos from our past build and test dive meetups, set it to music, and put it on Youtube.  We have  more footage than this and are working on a higher quality production, but I wanted to put a little teaser out there for some of what we have.  Check it out below and thanks for reading!

Again not so much

We tested the ROV again. It did not go well.

OpenROV, so pretty but still broken

At one point our motors wouldn't start at all. For no good reason. The ROV was running fine and a power cycle via the tether would not fix it. We had to pull batteries out of at least one tube, wait and re-power the ROV to get the motors back. Maybe one of the tether-driven 12v switches got stuck off?

AustinROV team troubleshooting

A recurring problem was that the ROV would shut down after some driving in the water. A power cycle via the tether would not bring it back and we had to pull batteries out of the tubes before we could get it to turn on again. We think this is due to the overcurrent protection built into the TrustFire batteries. The limit is 7.5 amps per tube and the ROV motors are rated for 9.5 amps each. This is why the OpenROV folks moved to a different battery type.

ROV reading No Diving sign

The ROV camera gave up near the end of our testing. We're still not sure what is up with that. No combination of battery pulling, rebooting or dashboard restarting would bring it back.

ROV looking ready to dive but no joy today

Thank you David H and Beth for the hospitality. Our tests didn't go very well but our tummies were sure happy!

Looks like we made it

Love's so strange

We fixed everything! Or, almost. Not quite everything on our checklist is complete but all of the "show stopping" bugs have been ironed out.

Full throttle reverse

downloading OpenROV software

Our main focus was the motors stopping mid-reverse at full power after only 2 seconds.


This was resolved by upgrading the OpenROV BeagleBone software to 2.5.1-rc3. The bug linked below describes the issue we've been fighting for way too long now. 

https://github.com/OpenROV/openrov-software/issues/239

Re-programming and re-calibrating

checking ESC programming

After flashing new software to the device, we noticed that two of the ESCs were acting erratic. With our new programming card, we were able to very quickly identify and correct the lost programming settings on the ESCs. Programming these Hobbywing ESCs by hand used to be a huge pain, waiting for the beeps of the menu repeating over and over again.

Now we can fix each ESC visually.

Re-calibration was also necessary because some motors didn't spin at the same speeds. We don't fully understand this one yet and are hopeful that it isn't necessary with each re-programming as it is a bit tedious.

So, if next time we power up the ROV and it doesn't work right, we'll try re-calibrating after a re-program of the ESCs. 

Dive, dive, dive!

A test dive is in order. And then, maybe a real one. WOOHOOO!

Elec-tricity, E-lec-tricity

Sorry we haven't posted much lately.  Things have been right troublin' with the ROV and the behavior problems it exhibits.  We're increasingly concerned that there's some sort of problem with the board, as we've now replaced two of the three ESC's that were misbehaving so badly, yet still seeing some of the same behaviors exhibited after the new ones were put in place.

One of the things I've enjoyed the most about the experience of building this piece of equipment is all of the interesting things I've learned along the way, but I've also tried to be candid about my amateur status (and that of some of the team) when it comes to the more advanced electronics on board, and I'm beginning to fear that in my amateurism, I (or someone, who knows) may have caused electrical damage at some point, which is causing these behavior issues.  It could also be that the problem was there from the start.  Whatever it is, we're looking at (possibly) a lot more testing and prep work before we get back in the water.  After the excitement of thinking we were finally there, it's a tough pill to swallow, but then again, things worth doing are rarely easy.  We'll keep testing away until it works, of that you can be sure.  I just don't know when that will be.

On a positive note (because I feel compelled to leave this post on a positive note) IF we end up having to replace lots of stuff - the good news is that after reading through the OpenROV forums, it would appear that we've spent about 10x more time building and fixing the ROV than most other groups have, so we're pretty darn practiced at this point with a lot of things that most teams only had to do once.  We could rebuild this thing in a day if we had to, so, in a strange way the problems we've had actually give me more confidence about the path forward than if things had gone smoothly.

So we're back to the testing bench and the multimeter for now, thanks for reading and we'll post again soon.

It's totally the hardware...unless it's not

Just a brief update today.  After our latest test flight, we had some positive signs, and some negative ones.  On the positive side, we've built a watertight craft with a bunch of lights, lasers, and cameras that all work now, so that's awesome!  But on the negative side, we've had yet more trouble with the motors, and with some bumps underwater causing the craft to die.  Like Danny mentioned in the last post, we're going to get our hot glue on to combat the issue of bumps knocking us out.  We're also going to be replacing these guys:

The ESC's we are using

We have noticed that on both dry runs and test flights, our motors are not responding correctly 100% of the time.  At times, the ESC's have lost their programming, and other times even with correct programming they act all crazy.  This team has put a lot of time and effort into this project, and the last thing any of us want is an ROV that only kind of works right and gets lost in a cave because the motors betray us.  So we're going to replace the ESC's and see how the motors act.  If they're still acting crazy, we need to look at the software, and we'll just have some spare ESC's out of it.


Another thing the team is doing is putting together a flight path for our first real world test flight (aka not in a swimming pool) that will be taking place at Spring Lake in San Marcos.  We're going to put our navigation and charting skills to the test in an environment where we can still see the ROV pretty clearly, since once we hit the caves, the difficulty will ramp up considerably.

Overall, while disappointing that we're still not quite there yet, it's also good that we're working our way through all of these mission-ending kinks now while we're diving in swimming pools, rather than 40 meters deep in a lightless cave. 
As always thanks for reading and don't forget to check us out on OpenExplorer where we'll be documenting our dives out in the world, and you can actually donate to the project there as well if you feel so inclined.  Cheers!