Project

General

Profile

MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot

Added by Anonymous almost 10 years ago

We just received one of our units back from the field and it hangs running a "mw.b" command in the u-boot param script. The system will lock up using DDR or internal ARM memory addresses in the mw.b command. If we boot off the mmc sd card with the same image, the mw.b command works fine. We then proceeded to check what was actually burned into NAND. We ran the following scripts on the unit to see if the MLO and u-boot.img's in NAND are the same as our released code on the mmc card. The output seen below shows that the two images are identical. We also checked the u-boot image in NAND by reading it into memory and running the imi command on it. The imi command says that the image is good.

The error is very consistent and we are unsure how to track this issue. If MLO and u-boot both appear correct, and match the released images, how could booting out of NAND vs. the MMC card cause this kind of failure? We are able to boot u-boot from NAND and stop it before we attempt to run our image, but issuing the mw.b command will still cause a lockup. Has Critical Link ever seen this type of behavior?

Any suggestions for what to test/ideas would be appreciated.

John Conover

----------------------------------------------
---Compare released images to NAND------------
U-Boot# mmc rescan
U-Boot# fatload mmc 0 0x8A000000 MLO
reading MLO

40291 bytes read
U-Boot# nandecc hw 2
HW ECC BCH8 Selected
U-Boot# nand read.i 0x8B000000 0x00000000 $filesize

NAND read: device 0 offset 0x0, size 0x9d63
40291 bytes read: OK
U-Boot# cmp.b 8A000000 8B000000 $filesize
Total of 40291 bytes were the same
U-Boot#
Total of 40291 bytes were the same
U-Boot# mmc rescan
U-Boot# fatload mmc 0 0x8C000000 u-boot.img
reading u-boot.img

381964 bytes read
U-Boot# nandecc hw 2
HW ECC BCH8 Selected
U-Boot# nand read.i 0x8D000000 0x00080000 $filesize

NAND read: device 0 offset 0x80000, size 0x5d40c
381964 bytes read: OK
U-Boot# cmp.b 8c000000 8d000000 $filesize
Total of 381964 bytes were the same
U-Boot#
--------------------------------------------------------

----------------------------------------------
--------Image check u-boot in NAND------------
U-Boot# nand read.i 0x8D000000 0x00080000 $filesize

NAND read: device 0 offset 0x80000, size 0x5d40c
381964 bytes read: OK
U-Boot# cmp.b 8c000000 8d000000 $filesize
Total of 381964 bytes were the same
U-Boot# nand read.i 0x8B000000 0x00080000 0x001e0000

NAND read: device 0 offset 0x80000, size 0x1e0000
1966080 bytes read: OK
U-Boot# imi 8c000000

  1. Checking Image at 8c000000 ...
    Legacy image found
    Image Name: U-Boot 21401-101.01.00.02-dirty
    Image Type: ARM U-Boot Firmware (uncompressed)
    Data Size: 381900 Bytes = 372.9 KiB
    Load Address: 80800000
    Entry Point: 00000000
    Verifying Checksum ... OK
    U-Boot#

Replies (27)

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

As added information, the mtest command works.

Commands that do work:
mtest 402fffe0 402ffffe A5A500FF
mtest 80000000 8d000000 A5A500FF

Commands that don't work:
mw.b 8d000000 A
mw.b 402fffe0 A
mw.w 402fffe0 A
mw.l 402fffe0 A

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

John,

What version of U-boot are you using? If you can include the full bootlog that would be helpful.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

Here is the boot sequence:
----------------------------------------------------------------
U-Boot# CCCCCCCC
U-Boot SPL 21401-101.01.00.02-dirty (Nov 06 2013 - 08:59:11)
Texas Instruments Revision detection unimplemented
MityARM335x profile 1 - Model No: 3352-GX-X27-RC Serial No: 134253
configuring for 256 MB DDR3
Critical Link AM335X Dev Kit -- NAND Page size = 2048k
Critical Link AM335X Dev Kit
PLL configuration complete

U-Boot 21401-101.01.00.02-dirty (Nov 06 2013 - 08:59:11)

I2C: ready
WARNING: Caches not enabled
NAND: HW ECC Hamming Code selected
256 MiB
MMC: OMAP SD/MMC: 0
Net: cpsw
Hit any key to stop autoboot: 0
Card did not respond to voltage select!
-------------------------------------------------

We have modified the original Makefile so that our Partnumber
would show on bootup. The original Makefile had:

VERSION = 2011
PATCHLEVEL = 09

Do you have a specific file that would carry your release number?

Thanks,
John C.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

There was a bug with the older u-boot where ecc was not enabled when the UBL loaded the u-boot image. This meant that bitflips could cause weird error messages. Seeing as you don't have any problems when the u-boot is loaded from mmc, I suspect this is what is happening.

You are likely seeing no difference between the images stored in nand vs mmc because when you run the nand read command ecc is enabled and is silently fixing the bitflips.

I recommend moving to the latest u-boot: u-boot-2013.10.
https://support.criticallink.com/redmine/projects/armc8-platforms/wiki/Das_U-Boot_Port
Note you can find the latest devkit sd card images in the am335x devkit files section:
https://support.criticallink.com/redmine/projects/am335x-devkit/files

I've attached an MLO and u-boot.img file from the latest devkit sd card for your convenience.
U-Boot mainline:
origin/u-boot-2013.10: d35e05656457c2d3715de8c3eb7c777375c160ae
u-boot config: mityarm335x_config

If you boot the attached u-boot files from mmc, you should be able to verify that there are bitflips by using the nand dump command to dump both the UBL and u-boot.img. It should print out when it fixes a bit flip.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

Hey Jonathan,
Something is very unhappy. Here is the boot:
CCCCCCCC
U-Boot SPL 2013.10-gd35e056 (Jun 03 2014 - 17:00:30)
MitySOM335x profile 1 - Model No: 3352-GX-X27-RC Serial No: 134253
Configuring for 256 MB DDR3 @ 400MHz
Critical Link AM335X Dev Kit -- NAND Page size = 2048k booting from dev 8
reading u-boot.img
reading u-boot.img

U-Boot 2013.10-gd35e056 (Jun 03 2014 - 17:00:30)

AM335X-GP rev 0
I2C: ready
DRAM: 256 MiB
WARNING: Caches not enabled
NAND: Using 2k bch8 layout
ECC Mode = 2 lo = 56/0
256 MiB
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
ECC: uncorrectable.
NAND read from offset 260000 failed -74
  • Warning - readenv() failed, using default environment

Net: cpsw:1 is connected to cpsw. Reconnecting to cpsw
cpsw
Hit any key to stop autoboot: 0
U-Boot#

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

Woops hangon didn't notice that you had a 3352 module. They have a larger nand and require a special u-boot build.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

U-Boot mainline:
origin/u-boot-2013.10: d35e05656457c2d3715de8c3eb7c777375c160ae
u-boot config: mityarm335x_4kpage_config

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

No nevermind, thats not correct. You do have a 256MB nand module. So the first u-boot should be correct.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

Those ecc errors are coming from reading the environment from nand. If I remember correctly the older u-boot saved the environment using a 1 bit ecc (hamming) instead of 8 bit ecc (bch-8) which the new u-boot uses for everything. You can ignore these errors for now.

But in the long run if you save the environment from the new u-boot then it won't have these problems.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

So let me try and clarify the issue. We have been running our u-boot for about 10 months with no issues. Are you suggesting that a bit has gone bad and because the older MLO did not set ecc correctly, we are now seeing this strang behavior with the mw command?

Thanks,
John C.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

That is what I'm suggesting. You can verify if this is the issue by running the 'nand dump' or 'nand read' command and looking for any reported fixed bitflips.

Note you can try this from linux as well. The -n option should allow you to disable ecc when reading the image. So you could do the following:

root@mityarm-335x ~ $ cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00800000 00010000 "NOR User Defined" 
mtd1: 00020000 00020000 "SPL" 
mtd2: 00020000 00020000 "SPL.backup1" 
mtd3: 00020000 00020000 "SPL.backup2" 
mtd4: 00020000 00020000 "SPL.backup3" 
mtd5: 001e0000 00020000 "U-Boot" 
mtd6: 00020000 00020000 "U-Boot Env" 
mtd7: 00500000 00020000 "Kernel" 
mtd8: 0f880000 00020000 "File System" 
root@mityarm-335x ~ $ nanddump -f /tmp/u-boot.dmp /dev/mtd5 -n
Block size 131072, page size 2048, OOB size 64
Dumping data starting at 0x00000000 and ending at 0x001e0000...
root@mityarm-335x ~ $ nanddump -f /tmp/u-boot_corrected.dmp /dev/mtd5
ECC failed: 0
ECC corrected: 0
Number of bad blocks: 0
Number of bbt blocks: 0
Block size 131072, page size 2048, OOB size 64
Dumping data starting at 0x00000000 and ending at 0x001e0000...
root@mityarm-335x ~ $ cmp /tmp/u-boot.dmp /tmp/u-boot_corrected.dmp
root@mityarm-335x ~ $

For reference:

root@mityarm-335x ~ $ nanddump
Usage: nanddump [OPTIONS] MTD-device
Dumps the contents of a nand mtd partition.

           --help               Display this help and exit
           --version            Output version information and exit
-a         --forcebinary        Force printing of binary data to tty
-c         --canonicalprint     Print canonical Hex+ASCII dump
-f file    --file=file          Dump to file
-l length  --length=length      Length
-n         --noecc              Read without error correction
-N         --noskipbad          Read without bad block skipping
-o         --omitoob            Omit oob data
-b         --omitbad            Omit bad blocks from the dump
-p         --prettyprint        Print nice (hexdump)
-q         --quiet              Don't display progress and status messages
-s addr    --startaddress=addr  Start address

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

Hey Jonathan,
I don't have access to a booted Linux, only u-boot. I created a text script that runs the u-boot "nand dump <address>" from address 80000h to F0000h in increments of 800h = 2048 (page size.) The output is raw (see included file) and I forgot what I should be looking for. Do you know of any u-boot "nand dump" options I could use? I see a "nand bad" command, but its out seems to quick for having done any checking.

U-Boot# nand device

Device 0: nand0, sector size 128 KiB
Page size 2048 b
OOB size 64 b
Erase size 131072 b
U-Boot# nand bad

Device 0 bad blocks:
U-Boot#

Thanks,
John C.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

Try doing to following. It should print out bitflips if it finds them.

U-Boot# nand read $loadaddr 0x0 0x80000

NAND read: device 0 offset 0x0, size 0x80000
 524288 bytes read: OK
U-Boot# nand read $loadaddr 0x80000 0x1e0000

NAND read: device 0 offset 0x80000, size 0x1e0000
Fixing bitflip at location 19d
Fixing bitflip at location 1b6
 1966080 bytes read: OK
U-Boot# nand read $loadaddr 0x280000 0x500000

NAND read: device 0 offset 0x280000, size 0x500000
Fixing bitflip at location 13
Fixing bitflip at location 23
Fixing bitflip at location 4c
Fixing bitflip at location f2
 5242880 bytes read: OK

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

If you confirm there are bit flips you can then flash just the UBL into the NAND and see if it boots fine. You'll see something like the following during boot to show that its correcting the bit flips.

U-Boot SPL 2013.10-g6c6a700 (Nov 20 2013 - 16:56:02)
MityARM335x profile 1 - Model No: 3359-GX-226-RL Serial No: 135287
Old factory config found, configuring for 256 MB DDR2
Configuring for 256 MB DDR2 @ 266MHz
Critical Link AM335X Dev Kit -- NAND Page size = 2048k booting from dev 5
Using 2k bch8 layout
ECC Mode = 2 lo = 56/0
nand_spl_load_image: loading 2048 bytes from 80000 to 80800000
nand_spl_load_image: loading 378212 bytes from 80000 to 807fffc0
Fixing bitflip at location 19d
Fixing bitflip at location 1b6

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

Hey Jonathan,
Yes there is indeed an error. I'll try the newer MLO to see how that works.
Thanks,
John C.

U-Boot# nand read 8d000000 0 80000

NAND read: device 0 offset 0x0, size 0x80000
524288 bytes read: OK
U-Boot# nand read 8d000000 80000 1e0000

NAND read: device 0 offset 0x80000, size 0x1e0000
Fixing bitflip at location 69
1966080 bytes read: OK
U-Boot#

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

Thanks Jonathan,
It seems the u-boot nandecc command has changed. We used to run the following script for
burning all four MLO locations. When we ran the "nandecc hw 2" command it used to come back
with a "HW ECC BCH8 Selected" response. Now it is just kicks back a "nandecc disabled". If I
just run "nandecc", it returns a "nand ecc mode = HW BCH8". Is it always on by default now?
John C.

mmc rescan
mw.b 0x8D000000 0xff 0x20000
fatload mmc 0 0x8D000000 MLO
nandecc hw 2
nand erase 0x00000000 0x1e0000
nand write.i 0x8D000000 0x00000000 $filesize
nand write.i 0x8D000000 0x00020000 $filesize
nand write.i 0x8D000000 0x00040000 $filesize
nand write.i 0x8D000000 0x00060000 $filesize

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

Looks like we can't reach u-boot with the new MLO. We see the following and then it stops. If we plug in the
mmc, we are back. Any suggestions?

John C.

U-Boot SPL 2013.10-gd35e056 (Jun 03 2014 - 17:00:30)
MitySOM335x profile 1 - Model No: 3352-GX-X27-RC Serial No: 134253
Configuring for 256 MB DDR3 @ 400MHz
Critical Link AM335X Dev Kit -- NAND Page size = 2048k booting from dev 5
Using 2k bch8 layout
ECC Mode = 2 lo = 56/0
nand_spl_load_image: loading 2048 bytes from 80000 to 80800000
nand_spl_load_image: loading 204800 bytes from 80000 to 80800000

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

Yes we decided to store everything in nand using the correct ecc so we disabled the nandecc command. So you do not need to change ecc modes anymore in u-boot.

Odd I wasn't aware of any compatibility problems with loading the old u-boot from the new UBL but maybe thats whats happening.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

Booting from the mmc card now does not show that u-boot in NAND has any issues.

U-Boot# nand read 8d000000 80000 1e0000

NAND read: device 0 offset 0x80000, size 0x1e0000
1966080 bytes read: OK
U-Boot#

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

Not sure what happened, but even placing the original MLO back it will not boot:

U-Boot SPL 21401-101.01.00.02-dirty (Nov 06 2013 - 08:59:11)
Texas Instruments Revision detection unimplemented
MityARM335x profile 1 - Model No: 3352-GX-X27-RC Serial No: 134253
configuring for 256 MB DDR3
Critical Link AM335X Dev Kit -- NAND Page size = 2048k
Critical Link AM335X Dev Kit
PLL configuration complete
mkimage signature not found - ih_magic = ffffffff
Assuming u-boot.bin ..

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

Seems like when I used our NAND mlo/u-boot burn in scripts, I blasted over the old u-boot by erasing NAND from 0 to 0x1E0000.

Sorry for the distraction.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Anonymous almost 10 years ago

Hey Jonathan,
Can you tell me if I were to pull a full u-boot "tree" from your "u-boot-mityarm-335x.git/tree", which one of
the head labels should I use? u-boot-2013.10, AM335XPSP_04.06.00.03 or AM335XPSP_04.06.00.08 ?
I'll need the full source code to compare to what we have so I can do a risk assessment of upgrading. Last upgrade
I just used the original code and added in our SPI NOR flash as the chip was so new.

Thanks,
John C.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

You were likely using the code from the AM335XPSP_04.06.00.08 branch before and now should be using the u-boot-2013.10 branch. There may be alot of changes between the two branches.

RE: MityARM335x profile 1 - Model No: 3352-GX-X27-RC failing memory writes in u-boot - Added by Jonathan Cormier almost 10 years ago

Looks like both the 03 and 08 branches have the same 2011.09 date so harder to determine which branch of code you were using.

(1-25/27) Go to top
Add picture from clipboard (Maximum size: 1 GB)