Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade LXC to latest stable version #2

Closed
pdsouza opened this issue Sep 2, 2016 · 12 comments
Closed

Upgrade LXC to latest stable version #2

pdsouza opened this issue Sep 2, 2016 · 12 comments

Comments

@pdsouza
Copy link
Member

pdsouza commented Sep 2, 2016

We currently use v1.0.7 which is quite a bit behind now.

This should fix the first-boot race condtition where LXC segfaults sometimes on an incomplete /data/maru directory:

09-01 20:37:53.163   203   203 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
09-01 20:37:53.163   203   203 F DEBUG   : Build fingerprint: 'Android/maru_hammerhead/hammerhead:6.0.1/MMB29V/dev08240006:userdebug/test-keys'
09-01 20:37:53.163   203   203 F DEBUG   : Revision: '0'
09-01 20:37:53.163   203   203 F DEBUG   : ABI: 'arm'
09-01 20:37:53.163   203   203 F DEBUG   : pid: 7207, tid: 7207, name: perspectived  >>> /system/bin/perspectived <<<
09-01 20:37:53.163   203   203 F DEBUG   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
09-01 20:37:53.164   203   203 F DEBUG   :     r0 00000000  r1 00000000  r2 00000000  r3 00000000
09-01 20:37:53.164   203   203 F DEBUG   :     r4 00000001  r5 b6f2dbf4  r6 0000000f  r7 00000000
09-01 20:37:53.164   203   203 F DEBUG   :     r8 00000000  r9 b6e17594  sl 00000000  fp be9ce95c
09-01 20:37:53.165   203   203 F DEBUG   :     ip b6e177d0  sp be9ce8f8  lr b6de0225  pc b6dba8ac  cpsr 60010030
09-01 20:37:53.165   848  1792 W NativeCrashListener: Couldn't find ProcessRecord for pid 7207
09-01 20:37:53.171   203   203 F DEBUG   : 
09-01 20:37:53.171   203   203 F DEBUG   : backtrace:
09-01 20:37:53.171   203   203 E DEBUG   : AM write failed: Broken pipe
09-01 20:37:53.171   203   203 F DEBUG   :     #00 pc 000188ac  /system/lib/libc.so (strlen+71)
09-01 20:37:53.171   203   203 F DEBUG   :     #01 pc 0003e221  /system/lib/libc.so (setenv+60)
09-01 20:37:53.171   203   203 F DEBUG   :     #02 pc 0001e86c  /system/lib/liblxc.so.1.0.7 (lxc_init+836)
09-01 20:37:53.171   203   203 F DEBUG   :     #03 pc 00020df8  /system/lib/liblxc.so.1.0.7 (__lxc_start+60)
09-01 20:37:53.171   203   203 F DEBUG   :     #04 pc 000215c8  /system/lib/liblxc.so.1.0.7 (lxc_start+120)
09-01 20:37:53.171   203   203 F DEBUG   :     #05 pc 000632f8  /system/lib/liblxc.so.1.0.7
09-01 20:37:53.171   203   203 F DEBUG   :     #06 pc 000023b9  /system/bin/perspectived
09-01 20:37:53.171   203   203 F DEBUG   :     #07 pc 0000240f  /system/bin/perspectived
09-01 20:37:53.171   203   203 F DEBUG   :     #08 pc 00002491  /system/lib/libperspective.so (android::BnPerspectiveService::onTransact(unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+84)
09-01 20:37:53.171   203   203 F DEBUG   :     #09 pc 00019985  /system/lib/libbinder.so (android::BBinder::transact(unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+60)
09-01 20:37:53.171   203   203 F DEBUG   :     #10 pc 0001ec67  /system/lib/libbinder.so (android::IPCThreadState::executeCommand(int)+542)
09-01 20:37:53.171   203   203 F DEBUG   :     #11 pc 0001edbd  /system/lib/libbinder.so (android::IPCThreadState::getAndExecuteCommand()+64)
09-01 20:37:53.171   203   203 F DEBUG   :     #12 pc 0001ee21  /system/lib/libbinder.so (android::IPCThreadState::joinThreadPool(bool)+48)
09-01 20:37:53.171   203   203 F DEBUG   :     #13 pc 00001e37  /system/bin/perspectived (main+162)
09-01 20:37:53.171   203   203 F DEBUG   :     #14 pc 00017359  /system/lib/libc.so (__libc_init+44)
09-01 20:37:53.171   203   203 F DEBUG   :     #15 pc 00001ee0  /system/bin/perspectived
09-01 20:37:53.220   203   203 F DEBUG   : 
09-01 20:37:53.220   203   203 F DEBUG   : Tombstone written to: /data/tombstones/tombstone_01

This patch in the latest stable-1.0 should fix it: lxc/lxc@83593a1.
@pdsouza
Copy link
Member Author

pdsouza commented Jan 28, 2017 •
edited
Loading

Yikes, ran a quick test on bullhead after merging in 1.0.9 and it crashes the device without any logs. armhf / hammerhead works fine though... More investigation coming.

I did an strace lxc-start -n jessie and it hangs and reboots the phone right after this:

clone(child_stack=0x7fdd9f7310, flags=CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|CLONE_NEWPID|SIGCHLD) = 5004
faccessat(AT_FDCWD, "/proc/5004/ns", X_OK) = 0
openat(AT_FDCWD, "/proc/5004/ns/mnt", O_RDONLY|O_CLOEXEC) = 22
openat(AT_FDCWD, "/proc/5004/ns/pid", O_RDONLY|O_CLOEXEC) = 23
openat(AT_FDCWD, "/proc/5004/ns/uts", O_RDONLY|O_CLOEXEC) = 24
openat(AT_FDCWD, "/proc/5004/ns/ipc", O_RDONLY|O_CLOEXEC) = 25
close(19)                               = 0
faccessat(AT_FDCWD, "/sbin/newuidmap", X_OK) = -1 ENOENT (No such file or directory)
faccessat(AT_FDCWD, "/vendor/bin/newuidmap", X_OK) = -1 ENOENT (No such file or directory)
faccessat(AT_FDCWD, "/system/sbin/newuidmap", X_OK) = -1 ENOENT (No such file or directory)
faccessat(AT_FDCWD, "/system/bin/newuidmap", X_OK) = -1 ENOENT (No such file or directory)
faccessat(AT_FDCWD, "/system/xbin/newuidmap", X_OK) = -1 ENOENT (No such file or directory)
geteuid()                               = 0
write(20, "\0\0\0\0", 4)                = 4
read(20, "\1\0\0\0", 4)                 = 4
openat(AT_FDCWD, "/sys/fs/cgroup/freezer/lxc/jessie/tasks", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 19
write(19, "5004", 4)                    = 4
close(19)                               = 0
openat(AT_FDCWD, "/sys/fs/cgroup/devices/lxc/jessie/tasks", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 19
write(19, "5004", 4)                    = 4
close(19)                               = 0
openat(AT_FDCWD, "/dev/cpuset/lxc/jessie/tasks", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 19
write(19, "5004", 4)                    = 4
close(19)                               = 0
openat(AT_FDCWD, "/dev/cpuctl/lxc/jessie/tasks", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 19
write(19, "5004", 4)                    = 4
close(19)                               = 0
openat(AT_FDCWD, "/acct/lxc/jessie/tasks", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 19
write(19, "5004", 4)                    = 4
close(19)                               = 0
write(20, "\2\0\0\0", 4)                = 4
read(20, lxc-start: external/lxc/src/lxc/conf.c: mk_devtmpfs: 1364 /dev/.lxc is not setup - taking fallback

@pdsouza
Copy link
Member Author

pdsouza commented Jan 29, 2017

Git bisected the bad commit to lxc/lxc@6bbb810

@pdsouza
Copy link
Member Author

pdsouza commented Jan 31, 2017

With strace -f lxc-start -n jessie I can see what looks like the process that mounts stuff in the container hang when it opens the console:

...
[pid  5159] openat(21, "mnt", O_RDONLY|O_NOFOLLOW) = 22
[pid  5159] close(21)                   = 0
[pid  5159] openat(22, "sdcard", O_RDONLY|O_NOFOLLOW) = 21
[pid  5159] close(22)                   = 0
[pid  5159] mount("/sdcard", "/proc/self/fd/21", 0x7fe099844b, MS_BIND, NULL) = 0
[pid  5159] close(21)                   = 0
[pid  5159] statfs("/sdcard", {f_type=0x65735546, f_bsize=4096, f_blocks=6523662, f_bfree=6104582, f_bavail=6100486, f_files=1667904, f_ffree=1614347, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=1070}) = 0
[pid  5159] mount("/sdcard", "/data/maru/lxc/rootfs/mnt/sdcard", 0x7fe099844b, MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_BIND, NULL) = 0
[pid  5159] read(20, "", 4096)          = 0
[pid  5159] close(20)                   = 0
[pid  5159] umask(0111)                 = 0
[pid  5159] mknodat(AT_FDCWD, "/data/maru/lxc/rootfs/dev/null", S_IFCHR|0777, makedev(1, 3)) = 0
[pid  5159] mknodat(AT_FDCWD, "/data/maru/lxc/rootfs/dev/zero", S_IFCHR|0777, makedev(1, 5)) = 0
[pid  5159] mknodat(AT_FDCWD, "/data/maru/lxc/rootfs/dev/full", S_IFCHR|0777, makedev(1, 7)) = 0
[pid  5159] mknodat(AT_FDCWD, "/data/maru/lxc/rootfs/dev/urandom", S_IFCHR|0777, makedev(1, 9)) = 0
[pid  5159] mknodat(AT_FDCWD, "/data/maru/lxc/rootfs/dev/random", S_IFCHR|0777, makedev(1, 8)) = 0
[pid  5159] mknodat(AT_FDCWD, "/data/maru/lxc/rootfs/dev/tty", S_IFCHR|0777, makedev(5, 0)) = 0
[pid  5159] mknodat(AT_FDCWD, "/data/maru/lxc/rootfs/dev/console", S_IFCHR|0600, makedev(5, 1)) = 0
[pid  5159] umask(0)                    = 0111
[pid  5159] faccessat(AT_FDCWD, "/data/maru/lxc/rootfs/dev/console", F_OK) = 0
[pid  5159] newfstatat(AT_FDCWD, "/data/maru/lxc/rootfs/dev/console", {st_mode=S_IFCHR|0600, st_rdev=makedev(5, 1), ...}, 0) = 0
[pid  5159] fchmodat(AT_FDCWD, "/dev/pts/5", 020600) = 0
[pid  5159] openat(AT_FDCWD, "/data/maru/lxc/rootfs", O_RDONLY) = 20
[pid  5159] openat(20, "dev", O_RDONLY|O_NOFOLLOW) = 21
[pid  5159] close(20)                   = 0
[pid  5159] openat(21, "console", O_RDONLY|O_NOFOLLOW

@pdsouza
Copy link
Member Author

pdsouza commented Jan 31, 2017 •
edited
Loading

So on Android 6.0 /proc/last_kmsg is now /sys/fs/pstore/console-ramoops and we can see the actual crash:

...
[  289.238687] Unhandled fault: synchronous external abort (0x96000010) at 0xffffff800087a000
[  289.238853] Internal error: : 96000010 [#1] PREEMPT SMP
[  289.239003] CPU: 0 PID: 4928 Comm: lxc-start Not tainted 3.10.73-g84d48e8-00002-gfb7b9c7 #1
[  289.239083] task: ffffffc01f4635c0 ti: ffffffc05fe34000 task.ti: ffffffc05fe34000
[  289.239243] PC is at msm_hsl_startup+0xc8/0x2ac
[  289.239323] LR is at msm_hsl_startup+0xac/0x2ac
[  289.239468] pc : [<ffffffc000563cbc>] lr : [<ffffffc000563ca0>] pstate: 800001c5
[  289.239545] sp : ffffffc05fe37a20
[  289.239688] x29: ffffffc05fe37a20 x28: ffffffc05fe37b6c 
[  289.239916] x27: 0000000000028000 x26: ffffffc001347fd6 
[  289.240207] x25: ffffffc05e539000 x24: ffffffc05f623590 
[  289.240432] x23: ffffffc001911b50 x22: ffffffc066414318 
[  289.240722] x21: 0000000000000030 x20: ffffffc00102a888 
[  289.240946] x19: ffffffc001911a58 x18: 0000007f9a302060 
[  289.241235] x17: 0000007f9a42985c x16: ffffffc0002f9220 
[  289.241460] x15: 0000007f9a30c3dc x14: 0000000000000063 
[  289.241684] x13: 000000000000001b x12: 000000000000001a 
[  289.241974] x11: 0000000000000000 x10: 00000000000001c3 
[  289.242199] x9 : 000000000000001e x8 : ffffffc029651120 
[  289.242489] x7 : 0000000000000000 x6 : 0000000000000001 
[  289.242713] x5 : 000000000000ffff x4 : ffffffc00134a94c 
[  289.243005] x3 : 0000000000000000 x2 : ffffff800087a000 
[  289.243230] x1 : 0000000000000140 x0 : 0000000000000050 
[  289.243519] 
[  289.243597] Process lxc-start (pid: 4928, stack limit = 0xffffffc05fe34058)
[  289.243676] Call trace:
[  289.243757] [<ffffffc000563cbc>] msm_hsl_startup+0xc8/0x2ac
[  289.243838] [<ffffffc000558094>] uart_startup+0x78/0x1b0
[  289.243982] [<ffffffc0005589f0>] uart_open+0xe8/0x14c
[  289.244065] [<ffffffc00054e43c>] tty_open+0x31c/0x49c
[  289.244214] [<ffffffc0002fdd4c>] chrdev_open+0x10c/0x140
[  289.244297] [<ffffffc0002f7f34>] do_dentry_open+0x18c/0x248
[  289.244442] [<ffffffc0002f8d58>] finish_open+0x34/0x54
[  289.244524] [<ffffffc000305af8>] do_last.isra.34+0x83c/0x9fc
[  289.244605] [<ffffffc000305d70>] path_openat.isra.35+0xb8/0x3fc
[  289.244748] [<ffffffc0003060e8>] do_filp_open+0x34/0x8c
[  289.244828] [<ffffffc0002f9160>] do_sys_open+0x128/0x1c0
[  289.244972] [<ffffffc0002f922c>] SyS_openat+0xc/0x18
[  289.245053] Code: 9b007c42 b8626a83 f9400a62 8b030042 (88dffc42) 
[  289.245198] ---[ end trace aa160cd3fa20bf86 ]---
[  289.278550] Kernel panic - not syncing: Fatal exception
[  289.278742] CPU2: stopping
[  289.278860] 
[  289.278961] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D      3.10.73-g84d48e8-00002-gfb7b9c7 #1
[  289.279139] task: ffffffc0072f0000 ti: ffffffc0072ec000 task.ti: ffffffc0072ec000
[  289.279333] PC is at lpm_cpuidle_enter+0x450/0x4d0
[  289.279429] LR is at lpm_cpuidle_enter+0x43c/0x4d0
[  289.279524] pc : [<ffffffc0008a6c30>] lr : [<ffffffc0008a6c1c>] pstate: 80000145
[  289.279699] sp : ffffffc0072efe40
[  289.279792] x29: ffffffc0072efe40 x28: 0000000000000001 
[  289.280149] x27: ffffffc061f4bc18 x26: ffffffc0010009c0 
[  289.280425] x25: ffffffc000e50000 x24: ffffffc061f4bc18 
[  289.280781] x23: 00000000000003e8 x22: ffffffc06a6650a8 
[  289.281058] x21: ffffffc061d0d018 x20: 0000000000000000 
[  289.281416] x19: ffffffc0072ec000 x18: 0000007f8503ba00 
[  289.281692] x17: 0000007f880b91c4 x16: ffffffc000335c04 
[  289.282051] x15: 0000000000000000 x14: 0ffffffffffffffe 
[  289.282329] x13: 0000000000000030 x12: 0101010101010101 
[  289.282687] x11: 7f7f7f7f7f7f7f7f x10: 7757597353435251 
[  289.282965] x9 : ffffffc061f4bcac x8 : 000be1fe9319d3b2 
[  289.283324] x7 : 0000000000000018 x6 : 0000000034155596 
[  289.283602] x5 : 0000000003ed9585 x4 : 0000000000000121 
[  289.283878] x3 : 00ffffffffffffff x2 : 0000000225c17d03 
[  289.284238] x1 : 000000435955eb0f x0 : 0000000000001dc7 
[  289.284515] 
[  289.284515] PC: 0xffffffc0008a6b30:
[  289.284690] 6b30  8b1c141c52800021 97fed53ab9400b80 2a1c03e153001c1c 97fedea12a1403e0 f9464800d0002d60 34000261b9400801 11000421b9405261 f9401018b9005261
[  289.285696] 6b70  f9400303b4000118 f94007002a1403e1 d63f00602a1c03e2 b5ffff40f8410f00 51000400b9405260 f9400260b9005260 941681cc36080040 aa1a03e1aa1503e0
[  289.286616] 6bb0  528000232a1403e2 f94252a097fffe22 8b1b001bb9405661 39403360b9008fa1 b94492a035000080 5400008b6b00029f 910233a1d28000a0 b9400b6097e73c41
[  289.287696] 6bf0  7100041f51000800 97e8242d54000048 d2807d1797ee62cb 2a0103e252800001 97e68f542a0103e3 f9403fa197e725e2 9ad70800cb010000 d50342ffb9000ac0
[  289.288698] 6c30  b94006d6f946bb35 340002a0b9400aa0 b940526097e7ab23 b900526011000400 b4000115f94012b5 12800001f94002a3 2a1603e2f94006a0 f8410ea0d63f0060
[  289.289695] 6c70  b9405260b5ffff40 b900526051000400 36080040f9400260 97e7aabf94168193 a94153f32a1403e0 a94363f7a9425bf5 a94573fba9446bf9 d65f03c0a8c97bfd
[  289.290693] 6cb0  d65f03c052800140 7100103fa9be7bfd 910003fd53001c42 2a0103f3a90153f3 d0003d81540001c8 38734821913ea021 8b21886110000063 52800021d61f0020
[  289.291608] 6cf0  5280006114000002 14000005b9001001 14000003b900101f 52800013b900101f 2a1303e1f9400000 2a0003f497f1c1ec d0003d8134000120 913ec021b0005a20
[  289.292690] 
[  289.292690] LR: 0xffffffc0008a6b1c:
[  289.292785] 6b18  36080040f9400260 2a1403e0941681ed f94252a097fede95 8b1c141c52800021 97fed53ab9400b80 2a1c03e153001c1c 97fedea12a1403e0 f9464800d0002d60
[  289.293774] 6b58  34000261b9400801 11000421b9405261 f9401018b9005261 f9400303b4000118 f94007002a1403e1 d63f00602a1c03e2 b5ffff40f8410f00 51000400b9405260
[  289.294855] 6b98  f9400260b9005260 941681cc36080040 aa1a03e1aa1503e0 528000232a1403e2 f94252a097fffe22 8b1b001bb9405661 39403360b9008fa1 b94492a035000080
[  289.295772] 6bd8  5400008b6b00029f 910233a1d28000a0 b9400b6097e73c41 7100041f51000800 97e8242d54000048 d2807d1797ee62cb 2a0103e252800001 97e68f542a0103e3
[  289.296769] 6c18  f9403fa197e725e2 9ad70800cb010000 d50342ffb9000ac0 b94006d6f946bb35 340002a0b9400aa0 b940526097e7ab23 b900526011000400 b4000115f94012b5
[  289.297767] 6c58  12800001f94002a3 2a1603e2f94006a0 f8410ea0d63f0060 b9405260b5ffff40 b900526051000400 36080040f9400260 97e7aabf94168193 a94153f32a1403e0
[  289.298764] 6c98  a94363f7a9425bf5 a94573fba9446bf9 d65f03c0a8c97bfd d65f03c052800140 7100103fa9be7bfd 910003fd53001c42 2a0103f3a90153f3 d0003d81540001c8
[  289.299843] 6cd8  38734821913ea021 8b21886110000063 52800021d61f0020 5280006114000002 14000005b9001001 14000003b900101f 52800013b900101f 2a1303e1f9400000
[  289.300758] 6d18  2a0003f497f1c1ec d0003d8134000120 913ec021b0005a20 91004021913c9000 2a1403e32a1303e2 2a1403e094165652 a8c27bfda94153f3 a9bf7bfdd65f03c0
[  289.301754] 
[  289.301754] SP: 0xffffffc0072efd40:
[  289.301930] fd40  0000000000000121 0000000003ed9585 0000000034155596 0000000000000018 000be1fe9319d3b2 ffffffc061f4bcac 7757597353435251 7f7f7f7f7f7f7f7f
[  289.302926] fd80  0101010101010101 0000000000000030 0ffffffffffffffe 0000000000000000 ffffffc000335c04 0000007f880b91c4 0000007f8503ba00 ffffffc0072ec000
[  289.303923] fdc0  0000000000000000 ffffffc061d0d018 ffffffc06a6650a8 00000000000003e8 ffffffc061f4bc18 ffffffc000e50000 ffffffc0010009c0 ffffffc061f4bc18
[  289.304918] fe00  0000000000000001 ffffffc0072efe40 ffffffc0008a6c1c ffffffc0072efe40 ffffffc0008a6c30 0000000080000145 00000000000003e8 ffffffc061f4bc18
[  289.305912] fe40  ffffffc0072efed0 ffffffc0008a3478 ffffffc06a6650a8 0000000000000000 000000435955eaa6 0000000000000000 ffffffc061e70800 0000000000000000
[  289.306989] fe80  ffffffc0012eb7f8 000000000007d000 ffffffc000e52000 0000004000000000 ffffffc061f4bc8c 0000000000000003 ffffffc001d40000 000000435955eb0f
[  289.307989] fec0  ffffffc061e70800 0000000200000000 ffffffc0072eff10 ffffffc0008a35d4 ffffffc06a6650a8 0000000000000000 ffffffc000e50000 0000000068e53000
[  289.308987] ff00  ffffffc061e70800 0000000000000000 ffffffc0072eff60 ffffffc000204808 ffffffc0072ec000 ffffffc00180f938 ffffffc00180f938 ffffffc000e51000
[  289.309984] 
[  289.310081] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D      3.10.73-g84d48e8-00002-gfb7b9c7 #1
[  289.310179] Call trace:
[  289.310367] [<ffffffc000206b88>] dump_backtrace+0x0/0x250
[  289.310464] [<ffffffc000206de8>] show_stack+0x10/0x1c
[  289.310570] [<ffffffc000e3def8>] dump_stack+0x1c/0x28
[  289.310751] [<ffffffc00020d38c>] handle_IPI+0xfc/0x1ec
[  289.310850] [<ffffffc000200704>] gic_handle_irq+0x80/0xa8
[  289.311025] Exception stack(0xffffffc0072efcf0 to 0xffffffc0072efe10)
[  289.311123] fce0:                                     072ec000 ffffffc0 00000000 00000000
[  289.311303] fd00: 072efe40 ffffffc0 008a6c30 ffffffc0 80000145 00000000 01c2c338 ffffffc0
[  289.311483] fd20: 00001dc7 00000000 5955eb0f 00000043 25c17d03 00000002 ffffffff 00ffffff
[  289.311662] fd40: 00000121 00000000 03ed9585 00000000 34155596 00000000 00000018 00000000
[  289.311759] fd60: 9319d3b2 000be1fe 61f4bcac ffffffc0 53435251 77575973 7f7f7f7f 7f7f7f7f
[  289.311937] fd80: 01010101 01010101 00000030 00000000 fffffffe 0fffffff 00000000 00000000
[  289.312117] fda0: 00335c04 ffffffc0 880b91c4 0000007f 8503ba00 0000007f 072ec000 ffffffc0
[  289.312215] fdc0: 00000000 00000000 61d0d018 ffffffc0 6a6650a8 ffffffc0 000003e8 00000000
[  289.312393] fde0: 61f4bc18 ffffffc0 00e50000 ffffffc0 010009c0 ffffffc0 61f4bc18 ffffffc0
[  289.312571] fe00: 00000001 00000000 072efe40 ffffffc0
[  289.312669] [<ffffffc0002035a4>] el1_irq+0x64/0xd4
[  289.312853] [<ffffffc0008a3474>] cpuidle_enter_state+0x40/0xb8
[  289.312949] [<ffffffc0008a35d0>] cpuidle_idle_call+0xe4/0x13c
[  289.313128] [<ffffffc000204804>] arch_cpu_idle+0x8/0x20
[  289.313232] [<ffffffc000268640>] cpu_startup_entry+0x1c4/0x234
[  289.313411] [<ffffffc000e392d0>] secondary_start_kernel+0x108/0x118
[  289.313509] CPU3: stopping
[  289.313804] 
[  289.313919] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D      3.10.73-g84d48e8-00002-gfb7b9c7 #1
[  289.314098] task: ffffffc0072f0ac0 ti: ffffffc0072f8000 task.ti: ffffffc0072f8000
[  289.314214] PC is at lpm_cpuidle_enter+0x450/0x4d0
[  289.314390] LR is at lpm_cpuidle_enter+0x43c/0x4d0
[  289.314487] pc : [<ffffffc0008a6c30>] lr : [<ffffffc0008a6c1c>] pstate: 80000145
[  289.314663] sp : ffffffc0072fbe40
[  289.314758] x29: ffffffc0072fbe40 x28: 0000000000000001 
[  289.315038] x27: ffffffc061f4bc18 x26: ffffffc0010009c8 
[  289.315396] x25: ffffffc000e50000 x24: ffffffc061f4bc18 
[  289.315672] x23: 00000000000003e8 x22: ffffffc06a6740a8 
[  289.316031] x21: ffffffc061d0d018 x20: 0000000000000000 
[  289.316308] x19: ffffffc0072f8000 x18: 0000000000000010 
[  289.316668] x17: 0000007f9a470af4 x16: ffffffc000316324 
[  289.316945] x15: 0000000000000001 x14: 0ffffffffffffffe 
[  289.317303] x13: 0000000000000030 x12: 0101010101010101 
[  289.317579] x11: 7f7f7f7f7f7f7f7f x10: 7757597353435251 
[  289.317936] x9 : ffffffc061f4bcac x8 : 000be1fe9319d3b2 
[  289.318213] x7 : 0000000000000018 x6 : 0000000034155596 
[  289.318571] x5 : 0000000003ed9585 x4 : 0000000000000121 
[  289.318846] x3 : 00ffffffffffffff x2 : 0000000225c17d03 
[  289.319203] x1 : 0000004359b08779 x0 : 0000000000000695 
[  289.319482] 
[  289.319482] PC: 0xffffffc0008a6b30:
[  289.319577] 6b30  8b1c141c52800021 97fed53ab9400b80 2a1c03e153001c1c 97fedea12a1403e0 f9464800d0002d60 34000261b9400801 11000421b9405261 f9401018b9005261
[  289.320658] 6b70  f9400303b4000118 f94007002a1403e1 d63f00602a1c03e2 b5ffff40f8410f00 51000400b9405260 f9400260b9005260 941681cc36080040 aa1a03e1aa1503e0
[  289.321654] 6bb0  528000232a1403e2 f94252a097fffe22 8b1b001bb9405661 39403360b9008fa1 b94492a035000080 5400008b6b00029f 910233a1d28000a0 b9400b6097e73c41
[  289.322650] 6bf0  7100041f51000800 97e8242d54000048 d2807d1797ee62cb 2a0103e252800001 97e68f542a0103e3 f9403fa197e725e2 9ad70800cb010000 d50342ffb9000ac0
[  289.323645] 6c30  b94006d6f946bb35 340002a0b9400aa0 b940526097e7ab23 b900526011000400 b4000115f94012b5 12800001f94002a3 2a1603e2f94006a0 f8410ea0d63f0060
[  289.324721] 6c70  b9405260b5ffff40 b900526051000400 36080040f9400260 97e7aabf94168193 a94153f32a1403e0 a94363f7a9425bf5 a94573fba9446bf9 d65f03c0a8c97bfd
[  289.325721] 6cb0  d65f03c052800140 7100103fa9be7bfd 910003fd53001c42 2a0103f3a90153f3 d0003d81540001c8 38734821913ea021 8b21886110000063 52800021d61f0020
[  289.326716] 6cf0  5280006114000002 14000005b9001001 14000003b900101f 52800013b900101f 2a1303e1f9400000 2a0003f497f1c1ec d0003d8134000120 913ec021b0005a20
[  289.327716] 
[  289.327716] LR: 0xffffffc0008a6b1c:
[  289.327893] 6b18  36080040f9400260 2a1403e0941681ed f94252a097fede95 8b1c141c52800021 97fed53ab9400b80 2a1c03e153001c1c 97fedea12a1403e0 f9464800d0002d60
[  289.328890] 6b58  34000261b9400801 11000421b9405261 f9401018b9005261 f9400303b4000118 f94007002a1403e1 d63f00602a1c03e2 b5ffff40f8410f00 51000400b9405260
[  289.329888] 6b98  f9400260b9005260 941681cc36080040 aa1a03e1aa1503e0 528000232a1403e2 f94252a097fffe22 8b1b001bb9405661 39403360b9008fa1 b94492a035000080
[  289.330884] 6bd8  5400008b6b00029f 910233a1d28000a0 b9400b6097e73c41 7100041f51000800 97e8242d54000048 d2807d1797ee62cb 2a0103e252800001 97e68f542a0103e3
[  289.331879] 6c18  f9403fa197e725e2 9ad70800cb010000 d50342ffb9000ac0 b94006d6f946bb35 340002a0b9400aa0 b940526097e7ab23 b900526011000400 b4000115f94012b5
[  289.332871] 6c58  12800001f94002a3 2a1603e2f94006a0 f8410ea0d63f0060 b9405260b5ffff40 b900526051000400 36080040f9400260 97e7aabf94168193 a94153f32a1403e0
[  289.333865] 6c98  a94363f7a9425bf5 a94573fba9446bf9 d65f03c0a8c97bfd d65f03c052800140 7100103fa9be7bfd 910003fd53001c42 2a0103f3a90153f3 d0003d81540001c8
[  289.334858] 6cd8  38734821913ea021 8b21886110000063 52800021d61f0020 5280006114000002 14000005b9001001 14000003b900101f 52800013b900101f 2a1303e1f9400000
[  289.335859] 6d18  2a0003f497f1c1ec d0003d8134000120 913ec021b0005a20 91004021913c9000 2a1403e32a1303e2 2a1403e094165652 a8c27bfda94153f3 a9bf7bfdd65f03c0
[  289.336942] 
[  289.336942] SP: 0xffffffc0072fbd40:
[  289.337038] bd40  0000000000000121 0000000003ed9585 0000000034155596 0000000000000018 000be1fe9319d3b2 ffffffc061f4bcac 7757597353435251 7f7f7f7f7f7f7f7f
[  289.338037] bd80  0101010101010101 0000000000000030 0ffffffffffffffe 0000000000000001 ffffffc000316324 0000007f9a470af4 0000000000000010 ffffffc0072f8000
[  289.339116] bdc0  0000000000000000 ffffffc061d0d018 ffffffc06a6740a8 00000000000003e8 ffffffc061f4bc18 ffffffc000e50000 ffffffc0010009c8 ffffffc061f4bc18
[  289.340108] be00  0000000000000001 ffffffc0072fbe40 ffffffc0008a6c1c ffffffc0072fbe40 ffffffc0008a6c30 0000000080000145 00000000000003e8 ffffffc061f4bc18
[  289.341024] be40  ffffffc0072fbed0 ffffffc0008a3478 ffffffc06a6740a8 0000000000000000 0000004359b08710 0000000000000000 ffffffc061e70800 0000000000000000
[  289.342022] be80  ffffffc0012eb7f8 ffffffc000e50000 ffffffc000e52000 0000000000000003 ffffffc061f4bc8c 0000000000000003 ffffffc001d40000 0000004359b08779
[  289.343018] bec0  ffffffc061e70800 0000000300000000 ffffffc0072fbf10 ffffffc0008a35d4 ffffffc06a6740a8 0000000000000000 ffffffc000e50000 0000000068e62000
[  289.344093] bf00  ffffffc061e70800 0000000000000000 ffffffc0072fbf60 ffffffc000204808 ffffffc0072f8000 ffffffc00180f938 ffffffc00180f938 ffffffc000e51000
[  289.345089] 
[  289.345185] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D      3.10.73-g84d48e8-00002-gfb7b9c7 #1
[  289.345362] Call trace:
[  289.345470] [<ffffffc000206b88>] dump_backtrace+0x0/0x250
[  289.345648] [<ffffffc000206de8>] show_stack+0x10/0x1c
[  289.345753] [<ffffffc000e3def8>] dump_stack+0x1c/0x28
[  289.345853] [<ffffffc00020d38c>] handle_IPI+0xfc/0x1ec
[  289.346032] [<ffffffc000200704>] gic_handle_irq+0x80/0xa8
[  289.346126] Exception stack(0xffffffc0072fbcf0 to 0xffffffc0072fbe10)
[  289.346304] bce0:                                     072f8000 ffffffc0 00000000 00000000
[  289.346485] bd00: 072fbe40 ffffffc0 008a6c30 ffffffc0 80000145 00000000 01c2c338 ffffffc0
[  289.346584] bd20: 00000695 00000000 59b08779 00000043 25c17d03 00000002 ffffffff 00ffffff
[  289.346763] bd40: 00000121 00000000 03ed9585 00000000 34155596 00000000 00000018 00000000
[  289.346945] bd60: 9319d3b2 000be1fe 61f4bcac ffffffc0 53435251 77575973 7f7f7f7f 7f7f7f7f
[  289.347124] bd80: 01010101 01010101 00000030 00000000 fffffffe 0fffffff 00000001 00000000
[  289.347222] bda0: 00316324 ffffffc0 9a470af4 0000007f 00000010 00000000 072f8000 ffffffc0
[  289.347402] bdc0: 00000000 00000000 61d0d018 ffffffc0 6a6740a8 ffffffc0 000003e8 00000000
[  289.347582] bde0: 61f4bc18 ffffffc0 00e50000 ffffffc0 010009c8 ffffffc0 61f4bc18 ffffffc0
[  289.347679] be00: 00000001 00000000 072fbe40 ffffffc0
[  289.347857] [<ffffffc0002035a4>] el1_irq+0x64/0xd4
[  289.347961] [<ffffffc0008a3474>] cpuidle_enter_state+0x40/0xb8
[  289.348140] [<ffffffc0008a35d0>] cpuidle_idle_call+0xe4/0x13c
[  289.348239] [<ffffffc000204804>] arch_cpu_idle+0x8/0x20
[  289.348423] [<ffffffc000268640>] cpu_startup_entry+0x1c4/0x234
[  289.348520] [<ffffffc000e392d0>] secondary_start_kernel+0x108/0x118
[  289.348699] CPU1: stopping
[  289.348828] 
[  289.348944] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D      3.10.73-g84d48e8-00002-gfb7b9c7 #1
[  289.349124] task: ffffffc00729eb80 ti: ffffffc0072e8000 task.ti: ffffffc0072e8000
[  289.349319] PC is at lpm_cpuidle_enter+0x450/0x4d0
[  289.349415] LR is at lpm_cpuidle_enter+0x43c/0x4d0
[  289.349591] pc : [<ffffffc0008a6c30>] lr : [<ffffffc0008a6c1c>] pstate: 80000145
[  289.349686] sp : ffffffc0072ebe40
[  289.349862] x29: ffffffc0072ebe40 x28: 0000000000000001 
[  289.350139] x27: ffffffc061f4bc18 x26: ffffffc0010009b8 
[  289.350416] x25: ffffffc000e50000 x24: ffffffc061f4bc18 
[  289.350773] x23: 00000000000003e8 x22: ffffffc06a6560a8 
[  289.351049] x21: ffffffc061d0d018 x20: 0000000000000000 
[  289.351406] x19: ffffffc0072e8000 x18: 00000000000001fd 
[  289.351682] x17: 0000007f9be9cc74 x16: ffffffc0002fa2a4 
[  289.352040] x15: 0000000000000002 x14: 0ffffffffffffffe 
[  289.352317] x13: 0000000000000030 x12: 0101010101010101 
[  289.352676] x11: 7f7f7f7f7f7f7f7f x10: 7757597353435251 
[  289.352952] x9 : ffffffc061f4bcac x8 : 000be1fe9319d3b2 
[  289.353310] x7 : 0000000000000018 x6 : 0000000034155596 
[  289.353588] x5 : 0000000003ed9585 x4 : 0000000000000121 
[  289.353945] x3 : 00ffffffffffffff x2 : 0000000225c17d03 
[  289.354221] x1 : 00000043575ee60e x0 : 0000000000009e8d 
[  289.354578] 
[  289.354578] PC: 0xffffffc0008a6b30:
[  289.354675] 6b30  8b1c141c52800021 97fed53ab9400b80 2a1c03e153001c1c 97fedea12a1403e0 f9464800d0002d60 34000261b9400801 11000421b9405261 f9401018b9005261
[  289.355680] 6b70  f9400303b4000118 f94007002a1403e1 d63f00602a1c03e2 b5ffff40f8410f00 51000400b9405260 f9400260b9005260 941681cc36080040 aa1a03e1aa1503e0
[  289.356679] 6bb0  528000232a1403e2 f94252a097fffe22 8b1b001bb9405661 39403360b9008fa1 b94492a035000080 5400008b6b00029f 910233a1d28000a0 b9400b6097e73c41
[  289.357759] 6bf0  7100041f51000800 97e8242d54000048 d2807d1797ee62cb 2a0103e252800001 97e68f542a0103e3 f9403fa197e725e2 9ad70800cb010000 d50342ffb9000ac0
[  289.358755] 6c30  b94006d6f946bb35 340002a0b9400aa0 b940526097e7ab23 b900526011000400 b4000115f94012b5 12800001f94002a3 2a1603e2f94006a0 f8410ea0d63f0060
[  289.359749] 6c70  b9405260b5ffff40 b900526051000400 36080040f9400260 97e7aabf94168193 a94153f32a1403e0 a94363f7a9425bf5 a94573fba9446bf9 d65f03c0a8c97bfd
[  289.360745] 6cb0  d65f03c052800140 7100103fa9be7bfd 910003fd53001c42 2a0103f3a90153f3 d0003d81540001c8 38734821913ea021 8b21886110000063 52800021d61f0020
[  289.361819] 6cf0  5280006114000002 14000005b9001001 14000003b900101f 52800013b900101f 2a1303e1f9400000 2a0003f497f1c1ec d0003d8134000120 913ec021b0005a20
[  289.362814] 
[  289.362814] LR: 0xffffffc0008a6b1c:
[  289.362909] 6b18  36080040f9400260 2a1403e0941681ed f94252a097fede95 8b1c141c52800021 97fed53ab9400b80 2a1c03e153001c1c 97fedea12a1403e0 f9464800d0002d60
[  289.363911] 6b58  34000261b9400801 11000421b9405261 f9401018b9005261 f9400303b4000118 f94007002a1403e1 d63f00602a1c03e2 b5ffff40f8410f00 51000400b9405260
[  289.364989] 6b98  f9400260b9005260 941681cc36080040 aa1a03e1aa1503e0 528000232a1403e2 f94252a097fffe22 8b1b001bb9405661 39403360b9008fa1 b94492a035000080
[  289.365987] 6bd8  5400008b6b00029f 910233a1d28000a0 b9400b6097e73c41 7100041f51000800 97e8242d54000048 d2807d1797ee62cb 2a0103e252800001 97e68f542a0103e3
[  289.366984] 6c18  f9403fa197e725e2 9ad70800cb010000 d50342ffb9000ac0 b94006d6f946bb35 340002a0b9400aa0 b940526097e7ab23 b900526011000400 b4000115f94012b5
[  289.367980] 6c58  12800001f94002a3 2a1603e2f94006a0 f8410ea0d63f0060 b9405260b5ffff40 b900526051000400 36080040f9400260 97e7aabf94168193 a94153f32a1403e0
[  289.369059] 6c98  a94363f7a9425bf5 a94573fba9446bf9 d65f03c0a8c97bfd d65f03c052800140 7100103fa9be7bfd 910003fd53001c42 2a0103f3a90153f3 d0003d81540001c8
[  289.370058] 6cd8  38734821913ea021 8b21886110000063 52800021d61f0020 5280006114000002 14000005b9001001 14000003b900101f 52800013b900101f 2a1303e1f9400000
[  289.371053] 6d18  2a0003f497f1c1ec d0003d8134000120 913ec021b0005a20 91004021913c9000 2a1403e32a1303e2 2a1403e094165652 a8c27bfda94153f3 a9bf7bfdd65f03c0
[  289.372045] 
[  289.372045] SP: 0xffffffc0072ebd40:
[  289.372220] bd40  0000000000000121 0000000003ed9585 0000000034155596 0000000000000018 000be1fe9319d3b2 ffffffc061f4bcac 7757597353435251 7f7f7f7f7f7f7f7f
[  289.373214] bd80  0101010101010101 0000000000000030 0ffffffffffffffe 0000000000000002 ffffffc0002fa2a4 0000007f9be9cc74 00000000000001fd ffffffc0072e8000
[  289.374208] bdc0  0000000000000000 ffffffc061d0d018 ffffffc06a6560a8 00000000000003e8 ffffffc061f4bc18 ffffffc000e50000 ffffffc0010009b8 ffffffc061f4bc18
[  289.375204] be00  0000000000000001 ffffffc0072ebe40 ffffffc0008a6c1c ffffffc0072ebe40 ffffffc0008a6c30 0000000080000145 00000000000003e8 ffffffc061f4bc18
[  289.376278] be40  ffffffc0072ebed0 ffffffc0008a3478 ffffffc06a6560a8 0000000000000000 00000043575ee5a6 0000000000000000 ffffffc061e70800 0000000000000000
[  289.377272] be80  ffffffc0012eb7f8 000000000007d000 ffffffc000e52000 0000004000000000 ffffffc061f4bc8c 0000000000000003 ffffffc001d40000 00000043575ee60e
[  289.378265] bec0  ffffffc061e70800 0000000100000000 ffffffc0072ebf10 ffffffc0008a35d4 ffffffc06a6560a8 0000000000000000 ffffffc000e50000 0000000068e44000
[  289.379262] bf00  ffffffc061e70800 0000000000000000 ffffffc0072ebf60 ffffffc000204808 ffffffc0072e8000 ffffffc00180f938 ffffffc00180f938 ffffffc000e51000
[  289.380338] 
[  289.380435] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D      3.10.73-g84d48e8-00002-gfb7b9c7 #1
[  289.380532] Call trace:
[  289.380720] [<ffffffc000206b88>] dump_backtrace+0x0/0x250
[  289.380818] [<ffffffc000206de8>] show_stack+0x10/0x1c
[  289.381006] [<ffffffc000e3def8>] dump_stack+0x1c/0x28
[  289.381105] [<ffffffc00020d38c>] handle_IPI+0xfc/0x1ec
[  289.381203] [<ffffffc000200704>] gic_handle_irq+0x80/0xa8
[  289.381380] Exception stack(0xffffffc0072ebcf0 to 0xffffffc0072ebe10)
[  289.381558] bce0:                                     072e8000 ffffffc0 00000000 00000000
[  289.381656] bd00: 072ebe40 ffffffc0 008a6c30 ffffffc0 80000145 00000000 01c2c338 ffffffc0
[  289.381835] bd20: 00009e8d 00000000 575ee60e 00000043 25c17d03 00000002 ffffffff 00ffffff
[  289.382014] bd40: 00000121 00000000 03ed9585 00000000 34155596 00000000 00000018 00000000
[  289.382111] bd60: 9319d3b2 000be1fe 61f4bcac ffffffc0 53435251 77575973 7f7f7f7f 7f7f7f7f
[  289.382290] bd80: 01010101 01010101 00000030 00000000 fffffffe 0fffffff 00000002 00000000
[  289.382469] bda0: 002fa2a4 ffffffc0 9be9cc74 0000007f 000001fd 00000000 072e8000 ffffffc0
[  289.382649] bdc0: 00000000 00000000 61d0d018 ffffffc0 6a6560a8 ffffffc0 000003e8 00000000
[  289.382747] bde0: 61f4bc18 ffffffc0 00e50000 ffffffc0 010009b8 ffffffc0 61f4bc18 ffffffc0
[  289.382924] be00: 00000001 00000000 072ebe40 ffffffc0
[  289.383102] [<ffffffc0002035a4>] el1_irq+0x64/0xd4
[  289.383208] [<ffffffc0008a3474>] cpuidle_enter_state+0x40/0xb8
[  289.383305] [<ffffffc0008a35d0>] cpuidle_idle_call+0xe4/0x13c
[  289.383484] [<ffffffc000204804>] arch_cpu_idle+0x8/0x20
[  289.383587] [<ffffffc000268640>] cpu_startup_entry+0x1c4/0x234
[  289.383765] [<ffffffc000e392d0>] secondary_start_kernel+0x108/0x118
[  289.383865] CPU4: stopping
[  289.384145] 
[  289.384288] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G      D      3.10.73-g84d48e8-00002-gfb7b9c7 #1
[  289.384538] task: ffffffc0072f1580 ti: ffffffc0072fc000 task.ti: ffffffc0072fc000
[  289.384690] PC is at lpm_cpuidle_enter+0x450/0x4d0
[  289.384936] LR is at lpm_cpuidle_enter+0x43c/0x4d0
[  289.385072] pc : [<ffffffc0008a6c30>] lr : [<ffffffc0008a6c1c>] pstate: 80000145
[  289.385314] sp : ffffffc0072ffe40
[  289.385447] x29: ffffffc0072ffe40 x28: 0000000000000001 
[  289.385948] x27: ffffffc061f4be18 x26: ffffffc0010009d0 
[  289.386337] x25: ffffffc000e50000 x24: ffffffc061f4be18 
[  289.386725] x23: 00000000000003e8 x22: ffffffc06a6830a8 
[  289.387116] x21: ffffffc061d0d818 x20: 0000000000000000 
[  289.387507] x19: ffffffc0072fc000 x18: 0000007f9a4eff50 
[  289.388002] x17: 0000007f9a471274 x16: ffffffc00022253c 
[  289.388386] x15: 0000007f9a4de000 x14: 0ffffffffffffffe 
[  289.388882] x13: 0000000000000030 x12: 0101010101010101 
[  289.389269] x11: 7f7f7f7f7f7f7f7f x10: 7757597353435251 
[  289.389764] x9 : ffffffc061f4beac x8 : 000be1fe9319d3b2 
[  289.390149] x7 : 0000000000000018 x6 : 0000000034155596 
[  289.390637] x5 : 0000000003ed9585 x4 : 0000000000000121 
[  289.391019] x3 : 00ffffffffffffff x2 : 0000000225c17d03 
[  289.391512] x1 : 0000004356116612 x0 : 000000000000f3ed 
[  289.391899] 
[  289.391899] PC: 0xffffffc0008a6b30:
[  289.392143] 6b30  8b1c141c52800021 97fed53ab9400b80 2a1c03e153001c1c 97fedea12a1403e0 f9464800d0002d60 34000261b9400801 11000421b9405261 f9401018b9005261
[  289.393533] 6b70  f9400303b4000118 f94007002a1403e1 d63f00602a1c03e2 b5ffff40f8410f00 51000400b9405260 f9400260b9005260 941681cc36080040 aa1a03e1aa1503e0
[  289.394917] 6bb0  528000232a1403e2 f94252a097fffe22 8b1b001bb9405661 39403360b9008fa1 b94492a035000080 5400008b6b00029f 910233a1d28000a0 b9400b6097e73c41
[  289.396302] 6bf0  7100041f51000800 97e8242d54000048 d2807d1797ee62cb 2a0103e252800001 97e68f542a0103e3 f9403fa197e725e2 9ad70800cb010000 d50342ffb9000ac0
[  289.397684] 6c30  b94006d6f946bb35 340002a0b9400aa0 b940526097e7ab23 b900526011000400 b4000115f94012b5 12800001f94002a3 2a1603e2f94006a0 f8410ea0d63f0060
[  289.399189] 6c70  b9405260b5ffff40 b900526051000400 36080040f9400260 97e7aabf94168193 a94153f32a1403e0 a94363f7a9425bf5 a94573fba9446bf9 d65f03c0a8c97bfd
[  289.400584] 6cb0  d65f03c052800140 7100103fa9be7bfd 910003fd53001c42 2a0103f3a90153f3 d0003d81540001c8 38734821913ea021 8b21886110000063 52800021d61f0020
[  289.401978] 6cf0  5280006114000002 14000005b9001001 14000003b900101f 52800013b900101f 2a1303e1f9400000 2a0003f497f1c1ec d0003d8134000120 913ec021b0005a20
[  289.403370] 
[  289.403370] LR: 0xffffffc0008a6b1c:
[  289.403616] 6b18  36080040f9400260 2a1403e0941681ed f94252a097fede95 8b1c141c52800021 97fed53ab9400b80 2a1c03e153001c1c 97fedea12a1403e0 f9464800d0002d60
[  289.405001] 6b58  34000261b9400801 11000421b9405261 f9401018b9005261 f9400303b4000118 f94007002a1403e1 d63f00602a1c03e2 b5ffff40f8410f00 51000400b9405260
[  289.406380] 6b98  f9400260b9005260 941681cc36080040 aa1a03e1aa1503e0 528000232a1403e2 f94252a097fffe22 8b1b001bb9405661 39403360b9008fa1 b94492a035000080
[  289.407766] 6bd8  5400008b6b00029f 910233a1d28000a0 b9400b6097e73c41 7100041f51000800 97e8242d54000048 d2807d1797ee62cb 2a0103e252800001 97e68f542a0103e3
[  289.409252] 6c18  f9403fa197e725e2 9ad70800cb010000 d50342ffb9000ac0 b94006d6f946bb35 340002a0b9400aa0 b940526097e7ab23 b900526011000400 b4000115f94012b5
[  289.410631] 6c58  12800001f94002a3 2a1603e2f94006a0 f8410ea0d63f0060 b9405260b5ffff40 b900526051000400 36080040f9400260 97e7aabf94168193 a94153f32a1403e0
[  289.412008] 6c98  a94363f7a9425bf5 a94573fba9446bf9 d65f03c0a8c97bfd d65f03c052800140 7100103fa9be7bfd 910003fd53001c42 2a0103f3a90153f3 d0003d81540001c8
[  289.413402] 6cd8  38734821913ea021 8b21886110000063 52800021d61f0020 5280006114000002 14000005b9001001 14000003b900101f 52800013b900101f 2a1303e1f9400000
[  289.414890] 6d18  2a0003f497f1c1ec d0003d8134000120 913ec021b0005a20 91004021913c9000 2a1403e32a1303e2 2a1403e094165652 a8c27bfda94153f3 a9bf7bfdd65f03c0
[  289.416280] 
[  289.416280] SP: 0xffffffc0072ffd40:
[  289.416415] fd40  0000000000000121 0000000003ed9585 0000000034155596 0000000000000018 000be1fe9319d3b2 ffffffc061f4beac 7757597353435251 7f7f7f7f7f7f7f7f
[  289.417812] fd80  0101010101010101 0000000000000030 0ffffffffffffffe 0000007f9a4de000 ffffffc00022253c 0000007f9a471274 0000007f9a4eff50 ffffffc0072fc000
[  289.419206] fdc0  0000000000000000 ffffffc061d0d818 ffffffc06a6830a8 00000000000003e8 ffffffc061f4be18 ffffffc000e50000 ffffffc0010009d0 ffffffc061f4be18
[  289.420595] fe00  0000000000000001 ffffffc0072ffe40 ffffffc0008a6c1c ffffffc0072ffe40 ffffffc0008a6c30 0000000080000145 00000000000003e8 ffffffc061f4be18
[  289.421980] fe40  ffffffc0072ffed0 ffffffc0008a3478 ffffffc06a6830a8 0000000000000000 0000004356116575 0000000000000000 ffffffc061e70400 0000000000000000
[  289.423352] fe80  ffffffc0012eb7f8 000000000007d000 ffffffc000e52000 0000004000000000 ffffffc061f4be8c 0000000000000003 ffffffc001d40000 0000004356116612
[  289.424833] fec0  ffffffc061e70400 0000000400000000 ffffffc0072fff10 ffffffc0008a35d4 ffffffc06a6830a8 0000000000000000 ffffffc000e50000 0000000068e71000
[  289.426208] ff00  ffffffc061e70400 0000000000000000 ffffffc0072fff60 ffffffc000204808 ffffffc0072fc000 ffffffc00180f938 ffffffc00180f938 ffffffc000e51000
[  289.427583] 
[  289.427721] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G      D      3.10.73-g84d48e8-00002-gfb7b9c7 #1
[  289.427960] Call trace:
[  289.428106] [<ffffffc000206b88>] dump_backtrace+0x0/0x250
[  289.428353] [<ffffffc000206de8>] show_stack+0x10/0x1c
[  289.428497] [<ffffffc000e3def8>] dump_stack+0x1c/0x28
[  289.428744] [<ffffffc00020d38c>] handle_IPI+0xfc/0x1ec
[  289.428882] [<ffffffc000200704>] gic_handle_irq+0x80/0xa8
[  289.429128] Exception stack(0xffffffc0072ffcf0 to 0xffffffc0072ffe10)
[  289.429265] fce0:                                     072fc000 ffffffc0 00000000 00000000
[  289.429510] fd00: 072ffe40 ffffffc0 008a6c30 ffffffc0 80000145 00000000 01c2c338 ffffffc0
[  289.429759] fd20: 0000f3ed 00000000 56116612 00000043 25c17d03 00000002 ffffffff 00ffffff
[  289.429898] fd40: 00000121 00000000 03ed9585 00000000 34155596 00000000 00000018 00000000
[  289.430142] fd60: 9319d3b2 000be1fe 61f4beac ffffffc0 53435251 77575973 7f7f7f7f 7f7f7f7f
[  289.430389] fd80: 01010101 01010101 00000030 00000000 fffffffe 0fffffff 9a4de000 0000007f
[  289.430524] fda0: 0022253c ffffffc0 9a471274 0000007f 9a4eff50 0000007f 072fc000 ffffffc0
[  289.430770] fdc0: 00000000 00000000 61d0d818 ffffffc0 6a6830a8 ffffffc0 000003e8 00000000
[  289.431017] fde0: 61f4be18 ffffffc0 00e50000 ffffffc0 010009d0 ffffffc0 61f4be18 ffffffc0
[  289.431261] fe00: 00000001 00000000 072ffe40 ffffffc0
[  289.431401] [<ffffffc0002035a4>] el1_irq+0x64/0xd4
[  289.431545] [<ffffffc0008a3474>] cpuidle_enter_state+0x40/0xb8
[  289.431680] [<ffffffc0008a35d0>] cpuidle_idle_call+0xe4/0x13c
[  289.431926] [<ffffffc000204804>] arch_cpu_idle+0x8/0x20
[  289.432067] [<ffffffc000268640>] cpu_startup_entry+0x1c4/0x234
[  289.432202] [<ffffffc000e392d0>] secondary_start_kernel+0x108/0x118
[  289.432342] CPU5: stopping
[  289.432539] 
[  289.432742] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G      D      3.10.73-g84d48e8-00002-gfb7b9c7 #1
[  289.432995] task: ffffffc0072f2040 ti: ffffffc007300000 task.ti: ffffffc007300000
[  289.433258] PC is at lpm_cpuidle_enter+0x450/0x4d0
[  289.433394] LR is at lpm_cpuidle_enter+0x43c/0x4d0
[  289.433638] pc : [<ffffffc0008a6c30>] lr : [<ffffffc0008a6c1c>] pstate: 80000105
[  289.433772] sp : ffffffc007303e40
[  289.434014] x29: ffffffc007303e40 x28: 0000000000000001 
[  289.434401] x27: ffffffc061f4be18 x26: ffffffc0010009d8 
[  289.434903] x25: ffffffc000e50000 x24: ffffffc061f4be18 
[  289.435288] x23: 00000000000003e8 x22: ffffffc06a6920a8 
[  289.435780] x21: ffffffc061d0d818 x20: 0000000000000000 
[  289.436167] x19: ffffffc007300000 x18: 0000000000000000 
[  289.436553] x17: 0000000000000000 x16: 0000000000000000 
[  289.437048] x15: 0000000000000000 x14: 0000000000000000 
[  289.437435] x13: 0000000000000000 x12: 0000000004c5d83d 
[  289.437931] x11: 0000000000000000 x10: 0000000000000000 
[  289.438319] x9 : ffffffc061f4beac x8 : 000be1fe9319d3b2 
[  289.438816] x7 : 0000000000000018 x6 : 0000000034155596 
[  289.439209] x5 : 0000000003ed9585 x4 : 0000000000000121 
[  289.439707] x3 : 00ffffffffffffff x2 : 0000000225c17d03 
[  289.440092] x1 : 000000423ed3a0cc x0 : 0000000000487049 
[  289.440586] 
[  289.440586] PC: 0xffffffc0008a6b30:
[  289.440721] 6b30  8b1c141c52800021 97fed53ab9400b80 2a1c03e153001c1c 97fedea12a1403e0 f9464800d0002d60 34000261b9400801 11000421b9405261 f9401018b9005261
[  289.442124] 6b70  f9400303b4000118 f94007002a1403e1 d63f00602a1c03e2 b5ffff40f8410f00 51000400b9405260 f9400260b9005260 941681cc36080040 aa1a03e1aa1503e0
[  289.443624] 6bb0  528000232a1403e2 f94252a097fffe22 8b1b001bb9405661 39403360b9008fa1 b94492a035000080 5400008b6b00029f 910233a1d28000a0 b9400b6097e73c41
[  289.444996] 6bf0  7100041f51000800 97e8242d54000048 d2807d1797ee62cb 2a0103e252800001 97e68f542a0103e3 f9403fa197e725e2 9ad70800cb010000 d50342ffb9000ac0
[  289.446258] 6c30  b94006d6f946bb35 340002a0b9400aa0 b940526097e7ab23 b900526011000400 b4000115f94012b5 12800001f94002a3 2a1603e2f94006a0 f8410ea0d63f0060
[  289.447643] 6c70  b9405260b5ffff40 b900526051000400 36080040f9400260 97e7aabf94168193 a94153f32a1403e0 a94363f7a9425bf5 a94573fba9446bf9 d65f03c0a8c97bfd
[  289.449015] 6cb0  d65f03c052800140 7100103fa9be7bfd 910003fd53001c42 2a0103f3a90153f3 d0003d81540001c8 38734821913ea021 8b21886110000063 52800021d61f0020
[  289.450502] 6cf0  5280006114000002 14000005b9001001 14000003b900101f 52800013b900101f 2a1303e1f9400000 2a0003f497f1c1ec d0003d8134000120 913ec021b0005a20
[  289.451886] 
[  289.451886] LR: 0xffffffc0008a6b1c:
[  289.452018] 6b18  36080040f9400260 2a1403e0941681ed f94252a097fede95 8b1c141c52800021 97fed53ab9400b80 2a1c03e153001c1c 97fedea12a1403e0 f9464800d0002d60
[  289.453515] 6b58  34000261b9400801 11000421b9405261 f9401018b9005261 f9400303b4000118 f94007002a1403e1 d63f00602a1c03e2 b5ffff40f8410f00 51000400b9405260
[  289.454906] 6b98  f9400260b9005260 941681cc36080040 aa1a03e1aa1503e0 528000232a1403e2 f94252a097fffe22 8b1b001bb9405661 39403360b9008fa1 b94492a035000080
[  289.456287] 6bd8  5400008b6b00029f 910233a1d28000a0 b9400b6097e73c41 7100041f51000800 97e8242d54000048 d2807d1797ee62cb 2a0103e252800001 97e68f542a0103e3
[  289.457675] 6c18  f9403fa197e725e2 9ad70800cb010000 d50342ffb9000ac0 b94006d6f946bb35 340002a0b9400aa0 b940526097e7ab23 b900526011000400 b4000115f94012b5
[  289.459065] 6c58  12800001f94002a3 2a1603e2f94006a0 f8410ea0d63f0060 b9405260b5ffff40 b900526051000400 36080040f9400260 97e7aabf94168193 a94153f32a1403e0
[  289.460457] 6c98  a94363f7a9425bf5 a94573fba9446bf9 d65f03c0a8c97bfd d65f03c052800140 7100103fa9be7bfd 910003fd53001c42 2a0103f3a90153f3 d0003d81540001c8
[  289.461845] 6cd8  38734821913ea021 8b21886110000063 52800021d61f0020 5280006114000002 14000005b9001001 14000003b900101f 52800013b900101f 2a1303e1f9400000
[  289.463222] 6d18  2a0003f497f1c1ec d0003d8134000120 913ec021b0005a20 91004021913c9000 2a1403e32a1303e2 2a1403e094165652 a8c27bfda94153f3 a9bf7bfdd65f03c0
[  289.464581] 
[  289.464581] SP: 0xffffffc007303d40:
[  289.464820] 3d40  0000000000000121 0000000003ed9585 0000000034155596 0000000000000018 000be1fe9319d3b2 ffffffc061f4beac 0000000000000000 0000000000000000
[  289.466201] 3d80  0000000004c5d83d 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffffffc007300000
[  289.467583] 3dc0  0000000000000000 ffffffc061d0d818 ffffffc06a6920a8 00000000000003e8 ffffffc061f4be18 ffffffc000e50000 ffffffc0010009d8 ffffffc061f4be18
[  289.469064] 3e00  0000000000000001 ffffffc007303e40 ffffffc0008a6c1c ffffffc007303e40 ffffffc0008a6c30 0000000080000105 00000000000003e8 ffffffc061f4be18
[  289.470456] 3e40  ffffffc007303ed0 ffffffc0008a3478 ffffffc06a6920a8 0000000000000000 000000423ed39fc8 0000000000000000 ffffffc061e70400 0000000000000000
[  289.471846] 3e80  ffffffc0012eb7f8 ffffffc000e50000 ffffffc000e52000 0000000000000005 ffffffc061f4be8c 0000000000000003 ffffffc001d40000 000000423ed3a0cc
[  289.473233] 3ec0  ffffffc061e70400 0000000500000000 ffffffc007303f10 ffffffc0008a35d4 ffffffc06a6920a8 0000000000000000 ffffffc000e50000 0000000068e80000
[  289.474610] 3f00  ffffffc061e70400 0000000000000000 ffffffc007303f60 ffffffc000204808 ffffffc007300000 ffffffc00180f938 ffffffc00180f938 ffffffc000e51000
[  289.476107] 
[  289.476245] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G      D      3.10.73-g84d48e8-00002-gfb7b9c7 #1
[  289.476486] Call trace:
[  289.476631] [<ffffffc000206b88>] dump_backtrace+0x0/0x250
[  289.476770] [<ffffffc000206de8>] show_stack+0x10/0x1c
[  289.477022] [<ffffffc000e3def8>] dump_stack+0x1c/0x28
[  289.477161] [<ffffffc00020d38c>] handle_IPI+0xfc/0x1ec
[  289.477408] [<ffffffc000200704>] gic_handle_irq+0x80/0xa8
[  289.477542] Exception stack(0xffffffc007303cf0 to 0xffffffc007303e10)
[  289.477789] 3ce0:                                     07300000 ffffffc0 00000000 00000000
[  289.477927] 3d00: 07303e40 ffffffc0 008a6c30 ffffffc0 80000105 00000000 00e49154 ffffffc0
[  289.478172] 3d20: 00487049 00000000 3ed3a0cc 00000042 25c17d03 00000002 ffffffff 00ffffff
[  289.478417] 3d40: 00000121 00000000 03ed9585 00000000 34155596 00000000 00000018 00000000
[  289.478660] 3d60: 9319d3b2 000be1fe 61f4beac ffffffc0 00000000 00000000 00000000 00000000
[  289.478795] 3d80: 04c5d83d 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  289.479041] 3da0: 00000000 00000000 00000000 00000000 00000000 00000000 07300000 ffffffc0
[  289.479285] 3dc0: 00000000 00000000 61d0d818 ffffffc0 6a6920a8 ffffffc0 000003e8 00000000
[  289.479421] 3de0: 61f4be18 ffffffc0 00e50000 ffffffc0 010009d8 ffffffc0 61f4be18 ffffffc0
[  289.479666] 3e00: 00000001 00000000 07303e40 ffffffc0
[  289.479914] [<ffffffc0002035a4>] el1_irq+0x64/0xd4
[  289.480056] [<ffffffc0008a3474>] cpuidle_enter_state+0x40/0xb8
[  289.480305] [<ffffffc0008a35d0>] cpuidle_idle_call+0xe4/0x13c
[  289.480443] [<ffffffc000204804>] arch_cpu_idle+0x8/0x20
[  289.480696] [<ffffffc000268640>] cpu_startup_entry+0x1c4/0x234
[  289.480832] [<ffffffc000e392d0>] secondary_start_kernel+0x108/0x118
[  290.721940] scm_call failed: func id 0x82000301, arginfo: 0x1, args: 0x1, 0xffffffc05fe37630, 0xffffffc000b385c0, 0xffffffc05fe37640, ret: -5, syscall returns: 0x0, 0x0, 0x0
[  290.722096] Rebooting in 5 seconds..
[  295.722437] Going down for restart now (pid: 4928, comm: lxc-start)
[  295.723443] qcom,qpnp-power-on qpnp-power-on-12: PMIC@SID2: configuring PON for reset
[  295.724791] qcom,qpnp-power-on qpnp-power-on-12: PMIC@SID2: configuring PON for reset
[  295.726026] qcom,qpnp-power-on qpnp-power-on-12: PMIC@SID2: configuring PON for reset
[  295.726800] Causing a watchdog bite!
[  295.727922] Wdog - STS: 0x1, CTL: 0x1, BARK TIME: 0x77fd3, BITE TIME: 0x1
No errors detected

@pdsouza
Copy link
Member Author

pdsouza commented Feb 1, 2017

This looks like a bullhead kernel issue to me. Something funky is going on with the openat() system call on the container's dev/console.

The fact that I can't reproduce this on armhf seems to confirm this as well.

@pdsouza
Copy link
Member Author

pdsouza commented Feb 2, 2017 •
edited
Loading

Disassembled the faulting instruction msm_hsl_startup+0xc8/0x2ac:

/var/maru/android_kernel_msm/drivers/tty/serial/msm_serial_hs_lite.c:1080

        vid = msm_hsl_port->ver_id;
        /* set automatic RFR level */
        data = msm_hsl_read(port, regmap[vid][UARTDM_MR1]);
ffffffc000563ca4 <msm_hsl_startup+0xb0> b9414a62        ldr     w2, [x19,#328]
ffffffc000563ca8 <msm_hsl_startup+0xb4> d2800a00        mov     x0, #0x50                       // #80
ffffffc000563cac <msm_hsl_startup+0xb8> 9b007c42        mul     x2, x2, x0
msm_hsl_read():
/var/maru/android_kernel_msm/drivers/tty/serial/msm_serial_hs_lite.c:188
                port->membase + off);
}
static inline unsigned int msm_hsl_read(struct uart_port *port,
                     unsigned int off)
{
        unsigned int v = le32_to_cpu((__force __le32)__raw_readl_no_log(
ffffffc000563cb0 <msm_hsl_startup+0xbc> b8626a83        ldr     w3, [x20,x2]
ffffffc000563cb4 <msm_hsl_startup+0xc0> f9400a62        ldr     x2, [x19,#16]
ffffffc000563cb8 <msm_hsl_startup+0xc4> 8b030042        add     x2, x2, x3
__raw_readl_no_log():
/var/maru/android_kernel_msm/arch/arm64/include/asm/io.h:77
}

static inline u32 __raw_readl_no_log(const volatile void __iomem *addr)
{
        u32 val;
        asm volatile("ldar %w0, [%1]" : "=r" (val) : "r" (addr));
ffffffc000563cbc <msm_hsl_startup+0xc8> 88dffc42        ldar    w2, [x2]
msm_hsl_read():
/var/maru/android_kernel_msm/drivers/tty/serial/msm_serial_hs_lite.c:190
                port->membase + off));
        __iormb();

So whatever address calculated in x2 and put out on the bus was invalid.

Link to source: https://android.googlesource.com/kernel/msm.git/+/android-msm-bullhead-3.10-marshmallow-dr1.6/drivers/tty/serial/msm_serial_hs_lite.c#1080

@pdsouza
Copy link
Member Author

pdsouza commented Feb 2, 2017 •
edited
Loading

I can confirm that swapping this line back to use normal mount fixes the crash: lxc/lxc@6bbb810#diff-9064c4ae46015f2c78f9deff397d67c7R1835

Maybe something weird is going on when dev/console is mounted on /proc/self/fd/ and then opened? Not obvious, but assuming a bug in the msm serial driver.

pdsouza added a commit that referenced this issue Feb 2, 2017
@pdsouza
Switch console setup from safe_mount back to mount
bd6ad6b 
The safe_mount call causes a serious kernel crash on bullhead so revert
back to a normal mount for now.

This will be reverted when all devices supported by Maru work with
safe_mount.

More info here: #2

Signed-off-by: Preetam D'Souza <[email protected]>
@pdsouza
Copy link
Member Author

pdsouza commented Feb 2, 2017

Patching the problematic line by dropping back to normal mount() for now: bd6ad6b

@pdsouza
Copy link
Member Author

pdsouza commented Feb 14, 2017

Confirmed that LXC v1.0.9 upgrades cleanly from v1.0.7 by installing Maru v0.4-dev as an update zip from v0.3 and running the old v0.3 container. Deleting /data/maru and syncing on the next boot works perfectly as well.

@pdsouza
Copy link
Member Author

pdsouza commented Feb 14, 2017

arm64-workaround has been fast-forward merged into master!

@pdsouza pdsouza closed this as completed Feb 14, 2017
@Aloento
Copy link

Aloento commented Feb 23, 2018 •
edited
Loading

Consider upgrading to LXC-1.0.9 or LXC 2？ @pdsouza
It seems that Maru is 1.0.9 now?

@pdsouza
Copy link
Member Author

pdsouza commented Feb 23, 2018

Yes, we use 1.0.9 now: 5fdb3ce.

LXC 1.0 will be supported upstream until 2019 so we have until then to upgrade to LXC 2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pdsouza @Aloento