Monday, December 22, 2014

These Poolish Games

Test Dive #5 is in the books.  

We set up an obstacle course underwater on Sunday to attempt to navigate and put ROV in the water.
Obstacle course sighted!
  We started flying, and immediately noticed that our balance in the water was way off, so we started working to adjust our weight distribution to get the ballast correct.  Turns out that split-shot sinkers from a fishing tackle box worked perfectly for this!
They have upgraded from catching sunfish in the river to ballasting an ROV - a huge promotion in a fishing weight's life, I'd think.  
Adjusting the weights 
Back in the water we went.  
Thanks to David H. for hosting us at his apartment pool again!

Some of the team chats with David while rebooting the ROV

Here's the part where things went wrong.  One of our pre-flight checks on our checklist (yes, we print out flight checklists every time we is important to be thorough!) is to check the seals on the tubes.  This is important because apparently you don't want water inside the main electronics tube of your ROV, who knew?  But what we didn't do was check the seals again after adjusting all the weights on the ROV.  So although newly balanced, we sprang a leak.  I think it came loose when we moved the tube around adjusting the weights.

So yes, you are correct, that is BAD NEWS BEARS.  
Bleh...water inside

It went down something like this.  I (James) was at the controls, wondering why we couldn't fly straight, just looping in circles.  Danny and Richard and Arthur were calling out to me to stop spinning in circles.  Someone noticed that there was fog in the main tube.  We pulled the ROV out of the pool and...well, yep, it's now about 5% full of water, enough to puddle up in the corner.  
The good news is that we never lost the video feed or the Cockpit from the ROV, so it still works.  The bad news is that obviously our dive was over for the day.

Drying off

Since we were gathered up and Arthur had brought some pro video equipment, we recorded some interviews we hope to incorporate later into a video we're making about the entire experience.  

The Interview that North Korea didn't want you to see

It takes a lot of gear to do things our way

Some positives we gain from this dive are that we had more control time underwater than ever before, thanks to our new LiPo batteries.  Next steps are to fix our trim issues so that we can fly straighter underwater, and to attempt the obstacle course again.  

Saturday, December 13, 2014

austinrov documentary teaser

None of us are filmmakers today but we're learning. The team worked on our interview questions and filming techniques today. Lots of footage was captured. Here we present a teaser, only a very small taste of the fun we had.

AustinROV teaser from Daniel Epperson on Vimeo.

We verified that the new batteries and camera servo were working as expected. One of the ESCs lost programming but it wasn't noticed or fixed. That might come back to haunt us later however it will be an easy fix.

Sunday, November 16, 2014

Some footage for your enjoyment

This past Friday, Danny E and I went to Spring Lake to scout out a potential test dive in a real world environment.  Unfortunately, the vegetation level and shallowness of the lake doesn't look great for using the ROV there, but we still managed to have a productive day.  We pieced together a bunch of the photos from our past build and test dive meetups, set it to music, and put it on Youtube.  We have  more footage than this and are working on a higher quality production, but I wanted to put a little teaser out there for some of what we have.  Check it out below and thanks for reading!

Friday, November 7, 2014

Again not so much

We tested the ROV again. It did not go well.

OpenROV, so pretty but still broken

At one point our motors wouldn't start at all. For no good reason. The ROV was running fine and a power cycle via the tether would not fix it. We had to pull batteries out of at least one tube, wait and re-power the ROV to get the motors back. Maybe one of the tether-driven 12v switches got stuck off?

AustinROV team troubleshooting

A recurring problem was that the ROV would shut down after some driving in the water. A power cycle via the tether would not bring it back and we had to pull batteries out of the tubes before we could get it to turn on again. We think this is due to the overcurrent protection built into the TrustFire batteries. The limit is 7.5 amps per tube and the ROV motors are rated for 9.5 amps each. This is why the OpenROV folks moved to a different battery type.

ROV reading No Diving sign

The ROV camera gave up near the end of our testing. We're still not sure what is up with that. No combination of battery pulling, rebooting or dashboard restarting would bring it back.

ROV looking ready to dive but no joy today

Thank you David H and Beth for the hospitality. Our tests didn't go very well but our tummies were sure happy!

Sunday, November 2, 2014

Looks like we made it

Love's so strange

We fixed everything! Or, almost. Not quite everything on our checklist is complete but all of the "show stopping" bugs have been ironed out.

Full throttle reverse

downloading OpenROV software
Our main focus was the motors stopping mid-reverse at full power after only 2 seconds.

This was resolved by upgrading the OpenROV BeagleBone software to 2.5.1-rc3. The bug linked below describes the issue we've been fighting for way too long now. 

Re-programming and re-calibrating

checking ESC programming
After flashing new software to the device, we noticed that two of the ESCs were acting erratic. With our new programming card, we were able to very quickly identify and correct the lost programming settings on the ESCs. Programming these Hobbywing ESCs by hand used to be a huge pain, waiting for the beeps of the menu repeating over and over again.

Now we can fix each ESC visually.

Re-calibration was also necessary because some motors didn't spin at the same speeds. We don't fully understand this one yet and are hopeful that it isn't necessary with each re-programming as it is a bit tedious.

So, if next time we power up the ROV and it doesn't work right, we'll try re-calibrating after a re-program of the ESCs. 

Dive, dive, dive!

A test dive is in order. And then, maybe a real one. WOOHOOO!

Thursday, October 9, 2014

Elec-tricity, E-lec-tricity

Sorry we haven't posted much lately.  Things have been right troublin' with the ROV and the behavior problems it exhibits.  We're increasingly concerned that there's some sort of problem with the board, as we've now replaced two of the three ESC's that were misbehaving so badly, yet still seeing some of the same behaviors exhibited after the new ones were put in place.

One of the things I've enjoyed the most about the experience of building this piece of equipment is all of the interesting things I've learned along the way, but I've also tried to be candid about my amateur status (and that of some of the team) when it comes to the more advanced electronics on board, and I'm beginning to fear that in my amateurism, I (or someone, who knows) may have caused electrical damage at some point, which is causing these behavior issues.  It could also be that the problem was there from the start.  Whatever it is, we're looking at (possibly) a lot more testing and prep work before we get back in the water.  After the excitement of thinking we were finally there, it's a tough pill to swallow, but then again, things worth doing are rarely easy.  We'll keep testing away until it works, of that you can be sure.  I just don't know when that will be.

On a positive note (because I feel compelled to leave this post on a positive note) IF we end up having to replace lots of stuff - the good news is that after reading through the OpenROV forums, it would appear that we've spent about 10x more time building and fixing the ROV than most other groups have, so we're pretty darn practiced at this point with a lot of things that most teams only had to do once.  We could rebuild this thing in a day if we had to, so, in a strange way the problems we've had actually give me more confidence about the path forward than if things had gone smoothly.

So we're back to the testing bench and the multimeter for now, thanks for reading and we'll post again soon.

Sunday, September 14, 2014

It's totally the hardware...unless it's not

Just a brief update today.  After our latest test flight, we had some positive signs, and some negative ones.  On the positive side, we've built a watertight craft with a bunch of lights, lasers, and cameras that all work now, so that's awesome!  But on the negative side, we've had yet more trouble with the motors, and with some bumps underwater causing the craft to die.  Like Danny mentioned in the last post, we're going to get our hot glue on to combat the issue of bumps knocking us out.  We're also going to be replacing these guys:

The ESC's we are using

We have noticed that on both dry runs and test flights, our motors are not responding correctly 100% of the time.  At times, the ESC's have lost their programming, and other times even with correct programming they act all crazy.  This team has put a lot of time and effort into this project, and the last thing any of us want is an ROV that only kind of works right and gets lost in a cave because the motors betray us.  So we're going to replace the ESC's and see how the motors act.  If they're still acting crazy, we need to look at the software, and we'll just have some spare ESC's out of it.

Another thing the team is doing is putting together a flight path for our first real world test flight (aka not in a swimming pool) that will be taking place at Spring Lake in San Marcos.  We're going to put our navigation and charting skills to the test in an environment where we can still see the ROV pretty clearly, since once we hit the caves, the difficulty will ramp up considerably.

Overall, while disappointing that we're still not quite there yet, it's also good that we're working our way through all of these mission-ending kinks now while we're diving in swimming pools, rather than 40 meters deep in a lightless cave. 
As always thanks for reading and don't forget to check us out on OpenExplorer where we'll be documenting our dives out in the world, and you can actually donate to the project there as well if you feel so inclined.  Cheers!

Sunday, September 7, 2014

Troubled so hard

We enthusiastically went for our third test dive, after having put away the ROV in 100% working order. Or so we thought. Our expedition was fraught with troubles.

outside underwater ROV view

One of our issues was the vertical propeller, which came unstuck from the motor bell and tried to escape out the finger guard.

vertical propeller escape attempt

Lag was also a huge issue. The ping times and responsiveness of the ROV vary wildly while in use, making it hard to predict and control. Speaking of control, I brought along a joystick which I think added much more confusion than it was worth. I do hope to set up a custom mapping for my many button joystick but that is going to be sidelined in favor of getting a reliable experience.

The ROV died at the bottom of the pool, ending the day.

dead, but with full batteries

This day ending loss of contact has happened to us once before, so we were already looking at the tether on the topside for failure points. The last time that this happened, it was the cat5 cable going into the beaglebone that shook loose. We fixed that with a piece of tape but wondered if maybe it came loose again. This time it was something on the tenda homeplug adapter, sandwiched between the beaglebone and the cape. A simple re-seat fixed the issue, but for how long?

Richard (and team) perplexed by the many issues

I'm thinking about hot-gluing the tenda homeplug adapter and cat5 connectors (at least) to keep them from bumping loose and killing the ROV.

Sunday, August 31, 2014

Open source rocks, except don't update it

it was dry, but still fun

Having a problem? Look at the source!

One of the awesome parts of working with an open source project is that you can go look at how it works in every aspect. Today I'm working on getting a joystick working with the OpenROV Cockpit and I think this is the file that will need to be modified to add support for a new joystick (and allow for key remapping). It turns out that unsupported controllers work pretty well.

Looking to upgrade? Don't do it!

If it ain't broke ... then don't fix it! I did it anyway. The script has now killed our ROV more than one time. I even left a comment in the file saying not to do it, but I still did it. Now the ROV Dashboard is broken, making it inoperable. Oops, again.

This opportunity forced me to pull down the newest code, fully abandoning the old mess of files at the filesystem level. This means losing my hard-coded IMU tweaks, maybe next time I'll find a way to make that adjustable (thinking cape atmega eeprom). The update should give us some new UI plugins anyway, so hey maybe it will be worth the pain.

That ended up being totally broken with "do not enter" warning signs. So I rolled back to the broken mess, then used git to roll back to a commit prior to the drastic changes to the dashboard. During the update, some nodejs stuff failed to compile, throwing a wrench into everything. In the end these few commands saved the day, for that issue. At least I got to keep my IMU hacks in place on the arduino cape.

tail -f /var/log/openrov.*
cd /opt/openrov/node_modules/
/opt/node/bin/npm install

After all this headache, the starboard motor lost its programming again and makes the tell-tale beeping/braking sound and drives erratically. I guess we'll have to re-program this when we go diving, once the ROV is powered up, at least until we're able to order and replace the ESC for that motor.

Wednesday, August 27, 2014

OpenExplorer AND A TEST DIVE

The founders of the OpenROV project contacted me a little while back about posting on their site,, to document our adventures with the ROV.  I was hesitant to post anything there until we had a working ROV, so for a long time I haven't moved on the idea.   For anyone that has been reading this blog for a while, you know this isn't out of lack of desire.  We have put so much work into this project, and have had our fair share of setbacks, so it hasn't always been easy to keep the fire burning on this. As someone new to the maker world, I wasn't sure what to expect but I definitely wasn't expecting this much work.   But our hard work has finally paid off.   I can now say that after tonight, I am so happy to announce that we are finally up and running!   We completed a full underwater test and everything on the ROV is working!   Lights, lasers, motors, all works.  The motor programming stuck.  The lasers toggled on and off accurately ( funny story, we are idiots and thought the lowecase l for lasers was a forward slash, so they were working the whole time we thought they were broken) and even the depth hold command is working accurately.  We are finally ready to rock!!!!!

Next step is to practice deploying and flying in a pool. Video of the ROV flying in action to come.  I just couldn't wait to share!  Really really exciting things to come now that we are up and running, thank you so much to our friends and family that have encouraged us along the way and to the new friends that have come along and followed our adventures!  So in conclusion, we are now live at on the site and will finally move from the building to adventuring phase.

Thanks for reading so far, and here's to great things to come!

Ready to dunk

Video feed is up

Here goes nothing...

There she blows!





Live feed from the camera after dunking.  Still working!

More testing.  Still working like a champ.


Using washers provided by Danny as ballast.  They work brilliantly, and are relatively easy to adjust.  Awesome!

One last shot of her in the water.  We're done!

Saturday, August 23, 2014

Lube something something

one must lube the tubes

The port battery tube wouldn't seat properly. It turns out that the o-ring was dried out. I lightly filed the edge on the tubes and caps and applied oil to the rings. 

Now both tubes are easier to open and close and the seals look much better.

Also pictured is the untested ballast correction on the starboard side. We hope to figure that out next week. 

Saturday, August 16, 2014

Grinding for mine

Tonight we spent time cleaning up the excess epoxy potting and re-testing the motors. Before grinding, these motors tested as electrically waterproof but that was before the propellers could be attached.

Richard trimming away epoxy potting

As long as we don't trim too much epoxy off the motors they should stay waterproof.

But they didn't. On one motor we lost all electrical insulation to the water.

electrically leaky motor

We hoped that touch-ups on these tiny spots on the motors will recover the electrical insulation and still allow the propeller bells to fit over the motors.

... And they did! The motors now show infinite electrical resistance to water and the propellers fit.

back side closeup of potted ROV motor

front side closeup of potted ROV motor

Thursday, August 7, 2014

New openrov video

It helps me stay focused and committed to see a reminder of what we're building and contributing to.

Monday, August 4, 2014

Excuse me while I drip this out

James' previous post was referring to this article regarding re-potting our brushless motors. So, did it work??

We're not sure yet, but it certainly got better.

If we were experienced scientists already we would have written down the readings we took prior to re-potting the motors. But we didn't. My vague recollection of that evening was "it should have said zero but didn't" and I was too excited about finding the problem to remember to save any data. We think we saw readings from 0.09 to 0.20 Mohm on all three leads of all three motors and on one of the battery leads.

First we had to peel and pry off all of the blue tape epoxy molds.

James de-casing the freshly potted motors

Then the epoxy required trimming as expected. We will have to touch this up later.

The epoxy missed some spots

Did we get better electrical results? Maybe! It seems like we did, anyway. Our numbers from this round were 5-15 Mohms. We think we can improve on this number with some epoxy touch-ups. 

We also tested the previously repaired battery tube wire which no longer shows leakage to water. Yay!

Saturday, July 26, 2014


Pot those motors right
Set them overnight

We decided to pull the motors off and do some work on them to better seal them against current leakage. The site had some other groups post about a method they used to pot the motors for saltwater use, so in the interest of a robust solution as well as the idea of possibly doing a saltwater flight someday, we got to work.

Here we go!

As you can see above, we pulled the brushless motors apart, and then wrapped them up in painter's tape, to make them into epoxy holding little buckets.  We filled the propeller holes with wax to keep those clear, and then Danny made a super high tech funnel out of tin foil in which we mixed and poured the 2-ton epoxy.

Goop Dreams

This was trickier than it looks, it really was.  The goal is to let that epoxy fill in every single gap around the wiring, but you have to take pains to keep this super viscous substance from dripping into the bearings (and all over yourself and your workbench for that matter.  Something I failed at.  My workbench is a minefield of epoxy drippings from this project now.)

Set time is 24-48 hours on the epoxy, and we did this on Thursday, so on Sunday I'm going to pull the tape and take a look.  The next step will be to bust the dremel out and smooth the epoxy down so that the motors fit back together again.  We've seen mention of folks on the OpenROV forums making new custom bushings for this process - this may be a direction we eventually have to take, I'm not sure yet.

Settle down, you.

We're really in the nitty-gritty of making the thing work now.  It's an interesting process to do a project like this - the excitement and anticipation of all the cool things we'll do with it gets run up against the reality that making something like this go is actually some hard work.  I've used computers and electronics my entire life, and have worked on computers professionally for a decade now, but only after building this thing have I truly gained a perspective of awe and appreciation for the electrical engineers in this world that make all the wonderful things in our lives just work.  And that's another thing I've loved about this project, and about doing science in general - I've learned things and gained perspective on things just by doing.  It gives me great hope for the future of this project and I am really excited to continue working on this ROV.  We hope to be back in the water soon, and the fun will really begin.  Until the next post, cheers and thanks for reading!

But first, lemme take a selfie

Wednesday, July 9, 2014

One does not simply dive into mordor...

The ROV ran into some problems during the test dive that we didn't encounter at home. As James hinted, there were some electrical shorts that we didn't previously notice or check for. Things had been going pretty well up to this point and it is only now that we realize the vast amount of effort required to make this thing go.

First, we discovered that the motors leak power into the water on all leads. Maybe some level is okay but we think that our readings were too high.

The water testing tank, donated from a different project
James happened to have a perfect water test tank. It was a 5 gallon bucket from a project that had holes cut in the side. These holes allowed us to bring the ROV connector outside of the bucket for poking and prodding.

See that tiny brown mark next to the motor? 
One of the wires in the harness was routed incorrectly. This put it too close to the motor bell and caused it to get sliced up. We think this might be wicking water into the battery tube causing the moisture that we observed. Hopefully we can fix that with a solder, hotglue and heatshrink (SHH) fixture.

James confirming the burned wire
In hindsight we could (should) have tested our motors submerged in water before potting the harness into the main tube. This way we could have tried a few times and made sure the wires were all sealed and long enough after being cut several times.

In any case it needs to come all the way apart and fortunately there is a small loop of extra wire for the entire harness. That will give us some slack to try fixing the seals on the motor solder joins. All nine motor wires need to be repaired, we think. The only external wires that passed the continuity test go into the IMU, indicating maybe the motors are the problem and not the SHH joints.

Tuesday, July 8, 2014

Mama didn't raise no pool

Hey everybody!

So after Danny's solo work on the ROV, the team decided it was time to do a water test.  Most of the team was able to get together the day after Independence Day at the home of some lovely friends of ours (thanks Hatches!) and test in their pool.

Perching peacefully before plunging into the clear blue

We booted her up and tested the camera, lights, motors - everything worked.  She's on and ready to go.  Check it out!

View of the pool from the onboard cam.  

We've got live video from the ROV and everything is in working order.  So here it is.  The moment we have all been waiting for.

The ROV is going under for her maiden voyage.  


Thar she blows

Woo!  We have live underwater video of the pool from the ROV!
I can tell you, the team was pretty excited.  We cheered and Danny flew her around the pool.  Awesome! But first thing we all noticed - the buoyancy wasn't quite right - she rose straight up, nose up, immediately after dunking.  It's important to balance the buoyancy of the ROV - flying up and down can burn through our battery needlessly, so we talked about ways to balance it out.

Pulled her back up - let's fix this buoyancy

And Richard contributed an interesting way of zeroing out that buoyancy:

Science can ball on a budget too.  Why not tape some rocks to the battery tubes?

So we taped the ROV down with some rocks to even it out and tried it again.

We toss it in and - Ruh Oh.  


Just kidding.  Did I scare you?  No, the rocks didn't crash our ROV.  They worked out eventually, although it took a few tries to get it right.  No - the latency we first noticed has now become much more pronounced, and motor control commands are either going sporadically or not at all.  The ROV DID fly around a bit, but never for long before shorting out.

We pulled the ROV out and checked for leaks, and saw none.  Although there was a slight bit of condensation inside one of the battery tubes, it was slight enough that we weren't convinced it was the problem.  

So sadly, we had to call an end to the test to come back and lab the ROV again to see what was up.

Next post, Danny will be explaining some problems we found with the ROV and what we need to do to fix them - check back soon for that story, more pics, and as always thanks for reading the blog!