Broken NavBoard?

Problems of AR Drone 1.0, Freeflight 1.
wiredrat
Ready for take off
Posts: 13
Joined: 25 Jan 2011, 09:46

Broken NavBoard?

Post by wiredrat » 25 Jan 2011, 10:15

Hi,

After a crash, that broke the central cross, my Ar Drone refused to fly. I can connect with AR.FreeFligth and got video from frontal camera, but it says "EMERGENCY UNKNOWN" (got it in spanish "PROBLEMA INDETERMINADO"). AR.FreeFligth don't show any motors information and is not aware of any sensors. Mainboard green led is turned on, but motors leds didn't lit up. Drone software is 1.4.7 and AR.FreeFligth 1.6.1.

I have some Linux skills so I telnet'd and try some things. At this moment, I can't show you the exact error, but I'll try to complete the information this evening.

dmesg show a kernel error related to parrot5 serial port:

Jan 1 00:00:17 myhost user.warn kernel: [ 1.652520] ------------[ cut here ]------------
Jan 1 00:00:17 myhost user.warn kernel: [ 1.656989] WARNING: at /home/fdhaeyer/parrot/workspace/ARDrone/all_sources/linux/head/Linux/kernel/linux/drivers/parrot/serial/parrot5.c:156 parrot5_serial_int+0x280/0x414()
Jan 1 00:00:17 myhost user.warn kernel: [ 1.672567] Modules linked in:
Jan 1 00:00:17 myhost user.warn kernel: [ 1.675566] [<c002a9fc>] (dump_stack+0x0/0x14) from [<c0038b48>] (warn_on_slowpath+0x4c/0x84)
Jan 1 00:00:17 myhost user.warn kernel: [ 1.684077] [<c0038afc>] (warn_on_slowpath+0x0/0x84) from [<c01d7b0c>] (parrot5_serial_int+0x280/0x414)
Jan 1 00:00:17 myhost user.warn kernel: [ 1.693470] r6:00000000 r5:00011000 r4:c03e286c
Jan 1 00:00:17 myhost user.warn kernel: [ 1.698057] [<c01d788c>] (parrot5_serial_int+0x0/0x414) from [<c005da34>] (handle_IRQ_event+0x44/0x84)
Jan 1 00:00:17 myhost user.warn kernel: [ 1.707351] [<c005d9f0>] (handle_IRQ_event+0x0/0x84) from [<c005f6f4>] (handle_level_irq+0xac/0x154)
Jan 1 00:00:17 myhost user.warn kernel: [ 1.716470] r7:00000002 r6:c7a99820 r5:00000005 r4:c035af40
Jan 1 00:00:17 myhost user.warn kernel: [ 1.722114] [<c005f648>] (handle_level_irq+0x0/0x154) from [<c0026048>] (__exception_text_start+0x48/0x64)
Jan 1 00:00:17 myhost user.warn kernel: [ 1.731766] r6:00000000 r5:c035af40 r4:00000005
Jan 1 00:00:17 myhost user.warn kernel: [ 1.736357] [<c0026000>] (__exception_text_start+0x0/0x64) from [<c00268a8>] (__irq_svc+0x48/0x8c)
NOTE: As stated, I don't have the real logs right now. This one is from http://forum.parrot.com/ardrone/en/viewtopic.php?id=288" onclick="window.open(this.href);return false;, but is pretty much the same error I've got. I'll give more info in a few hours.

And when strace'd program.elf found some timeouts when reading from ttyPA1. I thing ttyPA1 it is the NavBoard serial port interface and looks very much like a hardware problem. However, I can't determine for sure if the problem is in NavBoard or in mainboard UART interfarce. Could anyone help me identified the problem before I spent more money on this toy? I won't like buying a new NavBoard and find out that the problem is elsewhere...

Thanks!

scorpion2k
Ready for take off
Posts: 22
Joined: 30 Nov 2010, 00:55

Re: Broken NavBoard?

Post by scorpion2k » 25 Jan 2011, 11:28

Hi,
wiredrat wrote: And when strace'd program.elf found some timeouts when reading from ttyPA1. I thing ttyPA1 it is the NavBoard serial port interface and looks very much like a hardware problem. However, I can't determine for sure if the problem is in NavBoard or in mainboard UART interfarce. Could anyone help me identified the problem before I spent more money on this toy? I won't like buying a new NavBoard and find out that the problem is elsewhere...
ttyPA1 are the motors, not the navboard. Navboard is at ttyPA2. You should check if the motor connections work properly. Maybe your wires or connectors got damaged when your central cross was damaged.

Best regards

scorpion2k
Ready for take off
Posts: 22
Joined: 30 Nov 2010, 00:55

Re: Broken NavBoard?

Post by scorpion2k » 25 Jan 2011, 11:38

Oh, I forgot to mention. If you want to make sure that your Navboard works, try the following:

killall -9 program.elf
echo -e -n "\2" > /dev/ttyPA2
echo -e -n "\1" > /dev/ttyPA2
dd if=/dev/ttyPA2 bs=46 count=1 | hexdump -C


Repeat the last command a few times. You should get raw values of the navigation board. At offset 0x00, you should see the length of the navigation data (uint16, == 0x002C). At offset 0x02, a sequence counter is placed (uint16) which should be updated quite fast (~2ms).

Best regards

wiredrat
Ready for take off
Posts: 13
Joined: 25 Jan 2011, 09:46

Re: Broken NavBoard?

Post by wiredrat » 25 Jan 2011, 11:52

scorpion2k wrote:Hi,

ttyPA1 are the motors, not the navboard. Navboard is at ttyPA2. You should check if the motor connections work properly. Maybe your wires or connectors got damaged when your central cross was damaged.

Best regards
Thanks! I will check that later. However, yesterday remove mainboard and connect it again and nothing happend.

BTW, there are 2 connectors to the cross and the motors, one with red/black wires and other with colors wires. Which one is the ttyPA1? I guess that red/black wires is the power wires and the other one is the data port.

However, could a faulty connector lead to the kernel dump I've got? I'll try to read console from ttyPA0 (I saw that on your blog :)) and the tips on your last post to check that point. I suppose that if another port works, UART should be ok.

scorpion2k
Ready for take off
Posts: 22
Joined: 30 Nov 2010, 00:55

Re: Broken NavBoard?

Post by scorpion2k » 25 Jan 2011, 12:22

wiredrat wrote: BTW, there are 2 connectors to the cross and the motors, one with red/black wires and other with colors wires. Which one is the ttyPA1? I guess that red/black wires is the power wires and the other one is the data port.
Yes. Red/Black is power per motor and the other wires are for data. Each motor has Rx/Tx + 1 GPIO connected. Each time, the mainboad wants to communicate with one of the motors, the GPIO for the motor to communicate with is switched. But I haven't yet measured out the wires.
wiredrat wrote: However, could a faulty connector lead to the kernel dump I've got? I'll try to read console from ttyPA0 (I saw that on your blog :)) and the tips on your last post to check that point. I suppose that if another port works, UART should be ok.
As far as I have seen, warn_on_slowpath occurs if interrupts lasts too long (as you can see in backtrace, the warning comes from handle_IRQ_event). This MAY be related to a broken wire. But it can also be caused by broken balls under the P6 chip..

MAPGPS
Strange wobble
Posts: 201
Joined: 26 Oct 2010, 03:03

Re: Broken NavBoard?

Post by MAPGPS » 25 Jan 2011, 15:23

wiredrat wrote: dmesg show a kernel error related to parrot5 serial port:

Jan 1 00:00:17 myhost user.warn kernel: [ 1.652520] ------------[ cut here ]------------
Jan 1 00:00:17 myhost user.warn kernel: [ 1.656989] WARNING: at /home/fdhaeyer/parrot/workspace/ARDrone/all_sources/linux/head/Linux/kernel/linux/drivers/parrot/serial/parrot5.c:156 parrot5_serial_int+0x280/0x414()
Jan 1 00:00:17 myhost user.warn kernel: [ 1.672567] Modules linked in:
Jan 1 00:00:17 myhost user.warn kernel: [ 1.675566] [<c002a9fc>] (dump_stack+0x0/0x14) from [<c0038b48>] (warn_on_slowpath+0x4c/0x84)
Jan 1 00:00:17 myhost user.warn kernel: [ 1.684077] [<c0038afc>] (warn_on_slowpath+0x0/0x84) from [<c01d7b0c>] (parrot5_serial_int+0x280/0x414)
Jan 1 00:00:17 myhost user.warn kernel: [ 1.693470] r6:00000000 r5:00011000 r4:c03e286c
Jan 1 00:00:17 myhost user.warn kernel: [ 1.698057] [<c01d788c>] (parrot5_serial_int+0x0/0x414) from [<c005da34>] (handle_IRQ_event+0x44/0x84)
Jan 1 00:00:17 myhost user.warn kernel: [ 1.707351] [<c005d9f0>] (handle_IRQ_event+0x0/0x84) from [<c005f6f4>] (handle_level_irq+0xac/0x154)
Jan 1 00:00:17 myhost user.warn kernel: [ 1.716470] r7:00000002 r6:c7a99820 r5:00000005 r4:c035af40
Jan 1 00:00:17 myhost user.warn kernel: [ 1.722114] [<c005f648>] (handle_level_irq+0x0/0x154) from [<c0026048>] (__exception_text_start+0x48/0x64)
Jan 1 00:00:17 myhost user.warn kernel: [ 1.731766] r6:00000000 r5:c035af40 r4:00000005
Jan 1 00:00:17 myhost user.warn kernel: [ 1.736357] [<c0026000>] (__exception_text_start+0x0/0x64) from [<c00268a8>] (__irq_svc+0x48/0x8c)
NOTE: As stated, I don't have the real logs right now. This one is from http://forum.parrot.com/ardrone/en/viewtopic.php?id=288" onclick="window.open(this.href);return false;, but is pretty much the same error I've got. I'll give more info in a few hours.
I think this kernel error has no relation to your crash.
Mine has such kernel error too, but fly OK.
This kernel error should be a Parrot exsiting bug.

swekat
Battery Charged
Posts: 7
Joined: 01 Jan 2011, 03:13

Re: Broken NavBoard?

Post by swekat » 25 Jan 2011, 20:06

scorpion2k wrote:Oh, I forgot to mention. If you want to make sure that your Navboard works, try the following:

killall -9 program.elf
echo -e -n "\2" > /dev/ttyPA2
echo -e -n "\1" > /dev/ttyPA2
dd if=/dev/ttyPA2 bs=46 count=1 | hexdump -C


Repeat the last command a few times. You should get raw values of the navigation board. At offset 0x00, you should see the length of the navigation data (uint16, == 0x002C). At offset 0x02, a sequence counter is placed (uint16) which should be updated quite fast (~2ms).

Best regards
Hi !

Any more info on what is found from the dump of ttyPA2 ... such as battery status height etc ???

Would be lovely to be able to log navdata locally on the drone with a script that repeatedly polls /dev/ttyPA2 ...

Greets ..

/Swekat

scorpion2k
Ready for take off
Posts: 22
Joined: 30 Nov 2010, 00:55

Re: Broken NavBoard?

Post by scorpion2k » 25 Jan 2011, 20:37

swekat wrote: Any more info on what is found from the dump of ttyPA2 ... such as battery status height etc ???

Would be lovely to be able to log navdata locally on the drone with a script that repeatedly polls /dev/ttyPA2 ...
/Swekat
I published some information at my blog. But you won't be able to dump ttyPA2 as long as program.elf is running.

swekat
Battery Charged
Posts: 7
Joined: 01 Jan 2011, 03:13

Re: Broken NavBoard?

Post by swekat » 25 Jan 2011, 22:20

[quote="scorpion2kI published some information at my blog. But you won't be able to dump ttyPA2 as long as program.elf is running.[/quote]

Hi

I'm able to dump the data ok ... and as you say i see the 2c length indicator...even while program.elf is running.
Are you saying that the rest of the navdata structure is "inconsistent" while program.elf is active ???

swekat
Battery Charged
Posts: 7
Joined: 01 Jan 2011, 03:13

Re: Broken NavBoard?

Post by swekat » 25 Jan 2011, 22:27

[quote="scorpion2kI published some information at my blog. But you won't be able to dump ttyPA2 as long as program.elf is running.[/quote]

Ah.. found your blog and the datastructure ... good work .. initially i thought this should be mappable to a _navdata_demo_t structure or something but no such luck i guesss :) .. My aim is to try to log battery
altidude vx,vy,vz locally on the drone. Have tried thre geluar way with
listening on udp 5554 from a cprogram on the drone .. but no such luck !

/swekat

wiredrat
Ready for take off
Posts: 13
Joined: 25 Jan 2011, 09:46

Re: Broken NavBoard?

Post by wiredrat » 25 Jan 2011, 22:41

I could test my AR.Drone with scorpion2k advise.

I made a mistake when I said that ttyPA1 was giving problems. It is ttyPA2. I tried the scoprion2k commands and dd didn't got anything. So I figure that my Navboard is KO. Is there anything more I can do before buying for a new one?

Some detailed info:

Code: Select all

# strace -v -x -o /tmp/trace program.elf >/tmp/out   
# head -10 /tmp/out
Owner's MAC address is: 00:00:00:00:00:00
Clearing pairing rule
thread start
272.510776  NULL  6  784240  pic_read_select timeout
272.511633  NULL  6  0  version actuelle du PIC 0, version du PLF 4003a
272.606975  NULL  6  212672  icsp_decod_hex OK
272.630683  NULL  6  1076282856  pic_read_select timeout
272.631453  NULL  6  212672  lecture_calibration_pic:
272.632258  NULL  6  212672  erreur recuperation des données de calibration
!!! Emergency landing from /home/aferran/.ardrone/mykonos/ARDrone_Branch_20101015_1_4/Mykonos/Soft/Build/../../Soft/Toy//Os/elinux/Com/com_adc.c:372. Reason ID : 0 , timeout : 0
and from /tmp/trace:

Code: Select all

open("/dev/ttyPA2", O_RDWR|O_NOCTTY)    = 11                                         
ioctl(11, SNDCTL_TMR_TIMEBASE or TCGETS, {c_iflags=0, c_oflags=0x4, c_cflags=0x18b4, c_lflags=0xa30, c_line=0, c_cc[VMIN]=1, c_cc[VTIME]=0, c_cc="\x03\x1c\x
ioctl(11, SNDCTL_TMR_START or TCSETS, {c_iflags=0, c_oflags=0x4, c_cflags=0x18b4, c_lflags=0xa30, c_line=0, c_cc[VMIN]=1, c_cc[VTIME]=0, c_cc="\x03\x1c\x7f\
ioctl(11, TCFLSH, 0x1)                  = 0
write(11, "\x02", 1)                    = 1
ioctl(11, TCFLSH, 0)                    = 0                                                                                                              
[...snip...]                                                              
write(11, "\x05", 1)                    = 1
select(12, [11], NULL, NULL, {0, 20000}) = 0 (Timeout)   
Thanks all for your wise words!

MAPGPS
Strange wobble
Posts: 201
Joined: 26 Oct 2010, 03:03

Re: Broken NavBoard?

Post by MAPGPS » 26 Jan 2011, 04:06

wiredrat wrote: It is ttyPA2. I tried the scoprion2k commands and dd didn't got anything. So I figure that my Navboard is KO. Is there anything more I can do before buying for a new one?
You need to check the connector between NavBoard and MainBoard, there is a 2x4 Pin connector there.
swekat wrote: Any more info on what is found from the dump of ttyPA2 ... such as battery status height etc ???
Would be lovely to be able to log navdata locally on the drone with a script that repeatedly polls /dev/ttyPA2 ...
/Swekat
A sniffer tool Interceptty can help it:
http://www.sourcefiles.org/Networking/S ... 0.6.tar.gz" onclick="window.open(this.href);return false;
You can rename /dev/ttyPA2 as /dev/ttyPA2a, and run Interceptty to simulate a "fake" /dev/ttyPA2 for program.elf to use:)
Interceptty will bridge /dev/ttyPA2a with /dev/ttyPA2, and log everything received/sent.

wiredrat
Ready for take off
Posts: 13
Joined: 25 Jan 2011, 09:46

Re: Broken NavBoard?

Post by wiredrat » 26 Jan 2011, 11:15

MAPGPS wrote: You need to check the connector between NavBoard and MainBoard, there is a 2x4 Pin connector there.
Yep, checked and double-checked and it still no response frome navboard. IMHO it's crappy hardware :S
A flying machine prone to crash that break too easily... looks like a marketing decision. Ups, I'm getting a little paranoid :D
MAPGPS wrote: A sniffer tool Interceptty can help it:
http://www.sourcefiles.org/Networking/S ... 0.6.tar.gz" onclick="window.open(this.href);return false;
You can rename /dev/ttyPA2 as /dev/ttyPA2a, and run Interceptty to simulate a "fake" /dev/ttyPA2 for program.elf to use:)
Interceptty will bridge /dev/ttyPA2a with /dev/ttyPA2, and log everything received/sent.
Looks good. Did you compile it from ARM? Any library dependencies out of the Drone's SO?
I think that kernel use udev so renaming ttyPA2 could be as simple as adding some config at boot time.

MAPGPS
Strange wobble
Posts: 201
Joined: 26 Oct 2010, 03:03

Re: Broken NavBoard?

Post by MAPGPS » 26 Jan 2011, 15:02

wiredrat wrote:
MAPGPS wrote: A sniffer tool Interceptty can help it:
http://www.sourcefiles.org/Networking/S ... 0.6.tar.gz" onclick="window.open(this.href);return false;
You can rename /dev/ttyPA2 as /dev/ttyPA2a, and run Interceptty to simulate a "fake" /dev/ttyPA2 for program.elf to use:)
Interceptty will bridge /dev/ttyPA2a with /dev/ttyPA2, and log everything received/sent.
Looks good. Did you compile it from ARM? Any library dependencies out of the Drone's SO?
I think that kernel use udev so renaming ttyPA2 could be as simple as adding some config at boot time.
I have compiled Interceptty for ARM, and it can run on my AR.Drone to simulate a "fake" /dev/ttyPA2, logging NavData.
Unfortunately Interceptty has not yet implemented ioctl() calls, thus program.elf can not trigger NavData via ioctl(). Need to find a new serial port sniffer.

BTW. Regarding the kernel warning mesages, I tried to update drivers/parrot/serial/parrot5.c with WARN_ON_ONCE() commented out:

Code: Select all

	/* tty_flip_buffer_push doc say we can't call this from interrupt
	 * context, but other drivers do it
	 */
	//WARN_ON_ONCE(tty->low_latency == 1); //MAPGPS: to omit dmesg kenerl "WARNING:"

	spin_unlock(&port->lock);
	tty_flip_buffer_push(tty);
	spin_lock(&port->lock);
After loaded the new built kernel, the dmesg looks clean, no such kernel warning mesages anymore.

wiredrat
Ready for take off
Posts: 13
Joined: 25 Jan 2011, 09:46

Re: Broken NavBoard?

Post by wiredrat » 26 Jan 2011, 16:42

MAPGPS wrote: Unfortunately Interceptty has not yet implemented ioctl() calls, thus program.elf can not trigger NavData via ioctl(). Need to find a new serial port sniffer.
As far as I know, ioctl on ttyPA2 is used with TCSETS operation. This can be set using stty with aproppiate parameters. Not a dificult task. Other ioctl seen on ttyPA2 is TCFLSH (man tcflush(3)) which can affect to writings on device. If it really is an issue interceptty can be patched to do tcflush() every write on the real device file. More research should be done.
However, I didn't tried Interceptty, so I miss a lot of information (the fake device is a pipe or char special file?), and I'm sure that above is nonsense :)

Reading from ttyPA2 could provide some info, but navdata is well documented in SDK so it shouldn't be dificult to read. scorpion2k did a great work here. Motors interface ttyPA1 is more interesting because is not documented and knowing navdata and having motors control we can replace program.elf, and this is the "real hack" on this toy.

I'm so sad because of my partial dead drone :( I can't mess with it as much as I want.
MAPSGS wrote: BTW. Regarding the kernel warning mesages, I tried to update drivers/parrot/serial/parrot5.c with WARN_ON_ONCE() commented out:

Code: Select all

	/* tty_flip_buffer_push doc say we can't call this from interrupt
	 * context, but other drivers do it
	 */
	//WARN_ON_ONCE(tty->low_latency == 1); //MAPGPS: to omit dmesg kenerl "WARNING:"

	spin_unlock(&port->lock);
	tty_flip_buffer_push(tty);
	spin_lock(&port->lock);
After loaded the new built kernel, the dmesg looks clean, no such kernel warning mesages anymore.
Ummm, it is just a latency warning. Those backtrace scare me!

This is becoming OT on the "fixes" forum. May be we need to ask for a "Hacking" forum for this topics.

Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests