partition table conflicts when plugging SD card to newer ubuntu 15.04/14.04.2 kernel

[beta-tester]

according problems
https://www.raspberrypi.org/forums/viewtopic.php?f=91&t=111131
https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=105437

with plugging a NOOBS sd card to a newer ubuntu (debian) desktop PC/Laptop
i reported an issue report to ubuntu team
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1457526

and the problem seems to be the combination of existence of MSDOS partitiontable and RISCOS (partitiontable?) at the same time on the SD card.

i was looking into the code of NOOBS 1.4 and the RsicOS blob (the content of the file "riscos-boot.bin") will always be written to the SD card, even if it is not needed, because there is no RiscOS in the list of installed OS'es...
the newer kernel of ubuntu will catch that RiscOS blob/partitiontable first and mess up the detection of msdos partition tables...

do you see any chance to change that partition 'casino'?
is the RiscOS blob used for something else than a RISC OS, when it is installed?
maybe then only write the RiscOS blob to the SD card, if somebody installs a RISC OS, and elsewise fill the region with zeros.

what i was testing was,
i filled up the "riscos-boot.bin" file with zeros and forced with 'runinstaller' in cmdline recreation of partitions,
so the function InitDriveThread::writeRiscOSblob() will write only zeros to the SD card...
and that solved the problem of detecting the msdos partition table under newer ubuntu.
but it was only to be sure, that it is really the RiscOS blob.

with zeroing out riscos-boot.bin

dmesg:
[ 4119.677383] usb 6-1.7: new high-speed USB device number 8 using ehci-pci
[ 4119.771922] usb 6-1.7: New USB device found, idVendor=058f, idProduct=6335
[ 4119.771927] usb 6-1.7: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 4119.771930] usb 6-1.7: Product: Mass Storage Device
[ 4119.771932] usb 6-1.7: Manufacturer: Generic
[ 4119.771934] usb 6-1.7: SerialNumber: 058F011111B1
[ 4119.772463] usb-storage 6-1.7:1.0: USB Mass Storage device detected
[ 4119.772776] scsi host9: usb-storage 6-1.7:1.0
[ 4120.768974] scsi 9:0:0:0: Direct-Access     SD/MMC   Card  Reader     1.00 PQ: 0 ANSI: 0
[ 4120.769419] sd 9:0:0:0: Attached scsi generic sg3 type 0
[ 4121.004748] sd 9:0:0:0: [sdc] 31116288 512-byte logical blocks: (15.9 GB/14.8 GiB)
[ 4121.006953] sd 9:0:0:0: [sdc] Write Protect is off
[ 4121.006959] sd 9:0:0:0: [sdc] Mode Sense: 03 00 00 00
[ 4121.009098] sd 9:0:0:0: [sdc] No Caching mode page found
[ 4121.009104] sd 9:0:0:0: [sdc] Assuming drive cache: write through
[ 4121.033350]  sdc: sdc1 sdc2 < > sdc3
[ 4121.044793] sd 9:0:0:0: [sdc] Attached SCSI removable disk
[ 4121.640241] EXT4-fs (sdc3): mounted filesystem with ordered data mode. Opts: (null)

with normal riscos-boot.bin

dmesg:
[ 8519.331121] usb 6-1.7: new high-speed USB device number 13 using ehci-pci
[ 8519.425537] usb 6-1.7: New USB device found, idVendor=058f, idProduct=6335
[ 8519.425541] usb 6-1.7: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 8519.425544] usb 6-1.7: Product: Mass Storage Device
[ 8519.425546] usb 6-1.7: Manufacturer: Generic
[ 8519.425548] usb 6-1.7: SerialNumber: 058F011111B1
[ 8519.426076] usb-storage 6-1.7:1.0: USB Mass Storage device detected
[ 8519.426268] scsi host10: usb-storage 6-1.7:1.0
[ 8520.422700] scsi 10:0:0:0: Direct-Access     SD/MMC   Card  Reader     1.00 PQ: 0 ANSI: 0
[ 8520.423117] sd 10:0:0:0: Attached scsi generic sg3 type 0
[ 8520.658325] sd 10:0:0:0: [sdc] 31116288 512-byte logical blocks: (15.9 GB/14.8 GiB)
[ 8520.660444] sd 10:0:0:0: [sdc] Write Protect is off
[ 8520.660449] sd 10:0:0:0: [sdc] Mode Sense: 03 00 00 00
[ 8520.662539] sd 10:0:0:0: [sdc] No Caching mode page found
[ 8520.662542] sd 10:0:0:0: [sdc] Assuming drive cache: write through
[ 8520.684070]  sdc: [CUMANA/ADFS] sdc1 [ADFS] sdc1
[ 8520.695176] sd 10:0:0:0: [sdc] Attached SCSI removable disk

[lurch]

This is obviously a bug that I ought to (attempt to) fix, but how critical is it?
Is it only low-priority "this ought to work but it shouldn't", or is it high-priority "this prevents me from doing something that I really need to" ?
(it obviously only affects the NOOBS SD card when inserted into an Ubuntu PC, and has no affect on the Raspberry Pi itself)

[Rafiot]

In my case, it is a very annoying bug: I always setup my Raspberry Pi "offline" because I use them headless and I never have a screen with me.

Last time, I messed up the watchdog config over SSH (the rpi was basically always rebooting). The same kind of problem happened before with the network config. It is usually not a problem because I was expecting to be able to just change the config file from my linux box.

So for now, I don't use NOOBS because being able to just mount the key on linux is a requirement. But I agree that it is a quite specific case and I can simply use dd.

[beta-tester]

ah, it is a bug...
i was already wondering about the position where the RiscOS blob is written. to me it looks it is somewhere inside first partition (without corrupting the filesystem?)

i wouldn't say, it is a critical bug (in case, it will not corrupt the filesystem),
but it would be definitivly above normal.
i do a lot plugging the SD card to a PC to tweak some things, and i always have to remember and use the workaround woth fdisk to see the start of the partitions i want to mount and then mount the partitions with an offset...
indeed that is very annoying.

[maxnet]

i was already wondering about the position where the RiscOS blob is written. to me it looks it is somewhere inside first partition (without corrupting the filesystem?)

It is written right after the MBR in the spare space between the MBR and the first partition.
The proximity to the MBR was the reason the blob is written on initial NOOBS installation only, as a design choice at the time.

The way most flash storage works is that you cannot modify individual bytes.
If you want to write anything to the SD card, it has to read, erase and reprogram a larger block size area.
Wanted to keep writes to the area where the MBR lives to a minimum, to minimize the risk of the MBR getting corrupted, if a write goes wrong.

[lurch]

Thanks to the quirks of the MBR partitioning scheme, after NOOBS has done it's initial first-boot resizing, the MBR itself and the first partition never get modified. It's only files on the SETTINGS partition (mmcblk0p3) and all the logical partitions within the extended partition (mmcblk0p2) that get written to.
So even if a 'bad write' wiped out the entirety of mmcblk0p2 and mmcblk0p3, then NOOBS itself (stored on the mmcblk0p1 RECOVERY partition and never written to) would still be bootable, and allow you to install a fresh OS.

I've not looked into this problem in detail yet, but I wonder if it could be considered a bug that (the kernel used by) Ubuntu is looking for an ADFS partition-table first, and ignoring the completely valid MBR at the start of the disk? I believe if the MBR was read first, it would/should simply skip over the location of the ADFS partition table (aka "RiscOS blob").
...which I guess also explains why fdisk does work - because it looks for the MBR first.

[beta-tester]

yes, i made a mistake by resolving the definition of RISCOS_BLOB_SECTOR_OFFSET
it is indeed not inside in the first partition. it is in the not allocated region between the primary msdos partitiontable and the first partition...

my mistake was taking

#define RISCOS_OFFSET (1760)
#define RISCOS_SECTOR_OFFSET (RISCOS_OFFSET * 2048)

but it is

#define RISCOS_BLOB_SECTOR_OFFSET  (1)

so it will not corrupt the filesystem.

[lurch]

...okay, I think I've managed to track down the 'culprit' ;-)

As already (partially) mentioned, on a post-initial-boot NOOBS card, there's both an MBR partition table at bytes 1-512 inclusive, and an ADFS partition table at bytes 513-10240 inclusive (the 9728 bytes of riscos-boot.bin). Both partition tables are equally valid, with the MBR table describing the 'regular' linux partitions (Raspbian, OSMC, etc. as well as RECOVERY and SETTINGS), and the ADFS table describing the RISCOS partition (regardless of whether it's been installed yet or not). RISCOS doesn't understand MBR partitions, and so it always installs to a fixed location on the SD card.

In older versions of Ubuntu, running

grep ACORN_PARTITION /boot/config-`uname -r`

shows:

# CONFIG_ACORN_PARTITION_CUMANA is not set
# CONFIG_ACORN_PARTITION_ADFS is not set

And therefore the linux kernel ignores the ADFS table, and simply uses the MBR table (and all the regular partitions on your NOOBS card can be seen / mounted).

However in newer versions of Ubuntu, the same grep command shows:

CONFIG_ACORN_PARTITION_CUMANA=y
CONFIG_ACORN_PARTITION_ADFS=y

And as the comment at https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/block/partitions/check.c#n56 explains, the kernel partition-checking code looks for the Acorn partition table before looking for the msdos (MBR) partition table. Hence the cause of the problem as initially reported :-S

As both partition tables are equally valid, and RISCOS is unable to boot without the ADFS partition table, there's nothing that can be done from within NOOBS to fix this. So closing this for now as "wontfix".

However as a workaround you could either recompile your Ubuntu kernel to not include the ADFS / CUMANA options, or you could 'manually zero' the ADFS partition table with:
sudo dd if=/dev/zero of=/dev/sdd bs=512 conv=fsync seek=1 count=19
(replacing /dev/sdd with whatever device your SD card appears as). If you then remove and re-insert the SD card, Ubuntu should display all the regular partitions. Doing this will make RISCOS unbootable on that SD card though, unless you write riscos-boot.bin back to the card with:
sudo dd if=/path/to/riscos-boot.bin of=/dev/sdd bs=512 conv=fsync seek=1
(again replacing /dev/sdd as necessary).

As always, you need to be very careful what you're typing when using dd!

To check what's currently in the "ADFS area" of your SD card, run:
sudo dd if=/dev/sdd bs=512 skip=1 count=19 | md5sum
If it returns a162f87d777e2631927c2ead8ba75cf1 then the area is all-zeroes, if it returns 82a2ce581329b3eec1ba1afab1e11959 then it's a riscos-boot.bin from NOOBS v1.3.4 or earlier, and if it returns 6912dab36d2306dce07fa110fd077ea6 then it's a riscos-boot.bin from NOOBS v1.3.5 or later.

Even though I'm closing this bug, feel free to add additional comments or questions. And if you feel like pointing your forum threads and launchpad bug-report back to here, that'd be great :)

[beta-tester]

ok, no, i understand... then let it be like it is.

i read also that it seems, that adfs/cumana is also in newer updated Debian 8 kernels enabled...
is there an easy way to disable the adfs/cumana module without compiling a new kernel
(e.g. by using a blacklist)?
but unfortunately i did not found the names of these modules, what i have to put to a blacklist.

i know, it is a debian/ubuntu question - but hopefully you can help as well.

... thank you anyhow, for taking time to looking at.

[Rafiot]

CONFIG_ACORN_PARTITION_CUMANA=y
CONFIG_ACORN_PARTITION_ADFS=y

means is is compiled in the kernel, and not as a module, so there is no way you can blacklist it...

Anyway, good to know and to keep this bug as reference as it will avoid some headache to ubuntu/debian users.

[lurch]

Yup, and those config options are both boolean (yes/no) and not tristate (yes/no/module) http://cateee.net/lkddb/web-lkddb/ACORN_PARTITION_CUMANA.html
So a kernel (re)compilation is unfortunately the only way to turn them on or off.

And I also just found this bug-report which has the same symptom, but a different cause.
Heh, even 9 years ago this option was considered 'obsolete' http://debian-arm.debian.narkive.com/VkVbqy8f/disable-config-acorn-partition-cumana-in-debian-kernels
And see also http://lkml.iu.edu/hypermail/linux/kernel/0501.2/0543.html

So goodness knows why Ubuntu has now decided to enable this kernel option by default... :-(